AI Agents with JavaScript

A presentation from Belgrade.js meetup.

------------------------
How AI Agents work and how to build them with some JavaScript
------------------------

Did you know you already have all the skills you need to build an AI agent? You only need some JavaScript knowledge and an API key (optional but highly recommended).

AI agents have been a hot topic recently. Everybody talks about amazing, intelligent AI agents that often sound like popular science-fiction movies.

Let’s talk about AI agents in plain and simple language and learn how to build them using JavaScript and your favorite LLM.

Transcript

1. AI AGENTS with some JavaScript How work and how to build them
2. Have you heard about AI agents?
3. Yeah, the intelligent agents who will take your job in a few years!
4. I don't want to scare you, but…
5. "By2034,AI"agents"will: •Replace70%ofofficework(McKinsey) •Add$7trilliontotheglobaleconomy(Goldman) Mostjobswillbecomeobsolete. Here'swhatyouneedtoknow(&howyoushouldprepare)"! Somebodyontwitter (+McKinsey&Goldman)
8. So, welcome to my woodworking course!
9. AI AGENTS with some JavaScript
10. AI AGENTS with some JavaScript How work and how to build them
11. When people talk about AI agents, we often imagine:
12. But in reality, AI agents are more like:
13. Don't get me wrong. These agents can be extremely helpful!
14. How LLMs work But before we continue, let's see
19. Source: https://platform.openai.com/tokenizer
20. Prompts: - Prompts are instructions - You te! an LLM what you want, and it tries to reply based its training and your instructions - more clear instructions = better reply - LLM always answer, but not always based on truth
21. Prompts are just instructions
22. So, how do they work?
23. You give your instructions
25. You get some unexpected wisdom or hallucination
26. But how do LLMs know how to reply?
28. Tokens and vectors
29. @slobodan_ Source: https://cthiriet.com/blog/infinite-memory-llm
30. Anatomy of a prompt
31. The prompt is a set of textual instructions that fit LLM's context and other limitations
32. "Who is faster: Godzi!a or T-Rex?" A valid prompt
33. "Write a 500-word article about the bad influence of Amazon's RTO policy on Lambda cold starts" Also valid prompt
35. But LLMs are products that, like most other products, evolve with user needs and requests
36. System prompts
39. Some parts of your instructions might be more important than other or you might want to make them repeatable
40. Slobodan Stojanovic cofounder/CTO @ Vacation Tracker @slobodan_
41. Before we talk about AI agents, let me show you one more thing
42. Meet my "friend" Claude. I ask it many weird things all the time Way more weird than this one, trust me
43. This specific question is interesting because Claude can not answer it
45. Some other LLMs can answer this question! But I like Claude. Can I help it to answer?
49. By the way, we just created an AI agent!
50. I know ChatGPT can search the internet. But that's also an agent. It's just built into the ChatGPT product.
51. How AI agents work
52. LLMs can do and are good at this:
53. LLMs will always reply. Some replies might not be meaningful (hallucinations).
54. Agents are LLMs + something that provides missing information & capabilities
56. A "While" loop
57. An AI Agent is like a "while" loop that keeps asking available tools to provide additional information or capability until it has all it needs to complete the task or answer the question
58. The tool can be anything that provides the missing information or capabilities
59. I am the "tool" here!
60. But, while loops can be expensive!
61. Not because of the code complexity of a while loop, but because you invoke an LLM at least once in each iteration!
63. What does it mean to be "expensive" in this context? It depends on your use case! But be careful.
64. How to be careful: - Define spending limits - Make sure you do not iterate indefinitely (i.e., stop after N retries) - Use cheaper models for simple evaluations - Add monitoring and alarms
65. Where do we write these while loops?
66. Anywhere you want!
67. If you really really want, you can be that "while" loop
68. I am the "while loop" here!
69. But you can write your "while loop" anywhere you need it. This while loop can be in an app, in a terminal, on a server, in a browser, etc.
70. Just be careful not to expose your LLM secret keys and limit their usage because, remember, these while loops can be expensive.
71. How to write a while loop? - Define a system prompt with clear explanation of a! the tools you want to support (i.e., when and how to invoke them) - Ask an LLM to reply in the strict JSON format - Make sure you parse and validate reply correctly - Handle errors and have a limit on the number of iterations
72. LLMs are good at talking to humans, but these replies are not easy to parse in the code
74. You can ask an LLM to reply in the JSON format
75. User: // Some long instructions But always reply with valid JSON and nothing else! Here's your JSON: ```json { "some": "JSON", Assistant:
76. I SAID JSON ONLY!!! Works, sometimes
77. But there's something else you can do!
78. Write the beginning of the reply in the API request!
79. User: // Your instructions Answer with valid JSON and nothing else. { " Assistant: System: // Your system prompt
80. // Your instructions Answer with valid JSON and nothing else. { " // Your system prompt some": "valid", "JSON": true } User: Assistant: System:
81. It's good to understand how these "while loops" work. But, you don't really need to write your own while loop!
82. It's good to understand how these "while loops" work. But, you don't really need to write your own while loop!
83. Popular AI Agents tools & frameworks: - LlamaIndex - LangChain - AutoGen - Amazon Bedrock Agents - Many other alternatives…
84. https://ts.llamaindex.ai
85. LlamaIndex ≠ Meta Llama LLM
86. LlamaIndex Supported LLMs: - OpenAI LLms - Anthropic LLms - Groq LLMs - Llama2, Llama3, Llama3.1 LLMs - MistralAI LLMs - Fireworks LLMs - D"pS"k LLMs - ReplicateAI LLMs - TogetherAI LLMs - HuggingFace LLms - D"pInfra LLMs - Gemini LLMs
87. LlamaIndex recognized the AI assistant memory problem as an important thing to focus on
88. LLMs are limited by their context size
91. Luckily, the context of most LLMs is increasing fast
92. But we have more things we want to add to the context (such as Knowledge bases, documents, etc.)
93. RAG (Retrieval-Augmented Generation) Sounds complicated
98. Things to know about RAG - RAG is often explained in complicated terms, but it's a simple (and powerful) concept - You can use a vector database for RAG, but it's not necessary - You can store data almost anywhere, in vector DB, PostgreSQL, S3…
99. How does RAG work?
100. @slobodan_ Source: https://cthiriet.com/blog/infinite-memory-llm
101. 1. Split your Knowledge base into pieces and create vectors for each piece 2. Create vectors from user input 3. Do a vector search for the closest knowledge base matches to the user input 4. Add a knowledge base pieces to the prompt with an explanation/instructions 5. PROFIT How to use RAG:
102. But comparing vectors must be complicated!
104. However, RAG and vector search are topics for other presentations. Let's get back to agents…
105. Remember, LLMs are products that, like most other products, evolve with user needs and requests
106. So, now all major LLMs have built-in agent capabilities (or simply, they can use tools)
107. https://platform.openai.com/docs/guides/function-calling
108. https://docs.anthropic.com/en/docs/build-with-claude/tool-use/overview
109. Built-in tools pros: - Less errors (no n"d to force an LLM to return JSON) - No third-party tools - We! defined format
110. Built-in tools cons: - A bit harder to switch models (you n"d to write a sma! adapter/wrapper)
111. Let's build an AI agent!
114. We could build this in many different ways in production. For example, it can look similar to the following diagram
116. In production you need to think about: - Web application firewa! (WAF) with rate limiting - Error handling - Rate limits (for your app + LLMs + other services) - Monitoring - Conversation storage (i.e., DynamoDB) - And many other things
117. I'll show the most important parts of the code, only
118. Define available tools
120. Then you can invoke the LLM with the list of tools
122. The response
125. Then you send your API request, wait for the reply and do the following:
128. Agent response
130. That's it!
131. The complete code example is more complicated. I'll publish an article with a detailed code example and step-by-step guide soontm
132. A few other things to check
133. https://openai.com/index/new-tools-for-building-agents/
134. https://docs.anthropic.com/en/docs/agents-and-tools/mcp
135. https://slobodan.me/posts/5-prompt-engineering-tips-for-developers/
136. A quick summary
137. @slobodan_ • An AI Agent is like "while" loops with tools • You have all the skills you need to build tools • AI agents aren't scary, and they can be useful • Go, build agents, and have fun
138. https://slobodan.me @slobodan_