AI Agents — the new frontier for LLMs

Know Large Language Models at your fingertips? Mastering Retrieval Augmented Generation to help an LLM search your documents? It's time to dive into the wonderful world of intelligent agents, the next frontier for LLMs!

In this session, we will first define what agents are, or at least what makes a system "agentic". We will explain the limitations of LLMs. Then, through concrete examples, we will implement different agents in Java, using the LangChain4j framework, to illustrate some typical agent patterns and to understand how to go beyond a simple LLM call to obtain responses that meet the needs of your users, or even to trigger actions with the surrounding system.

But it’s not all we’ll learn about! An agent doesn’t live alone on a desert tropical island. Indeed it can communicate with other agents via tools that can be invoked thanks to the Model Context Protocol (MCP). They can also interact with other remote agents from other platforms and ecosystems, thanks to the Agent To Agent protocol (A2A).

Are you ready for the next hype on agents? Come and discover it in this session!

Transcript

1. AI Agents -—-—-—-—- the new frontier for LLMs Guillaume Laforge Developer Advocate glaforge glaforge.dev @glaforge @glaforge.dev @glaforge@uwyn.net
2. What is an AI agent? Agent design patterns Use case #1: Agentic RAG Use case #2: Sci-Fi story agent Model Context Protocol Agent Development Kit & the Agent 2 Agent protocol 01 02 03 04 05 06 📆 Agenda
3. LangChain4j
4. What is an AI Agent? 01 An LLM-powered System?
5. A general definition 🤖 AGENT 🧠 ENVIRONMENT sensors actuators perceive act Tools! An agent is a service that talks to an AI model to perform a goal-based operation using the tools and context it has.
6. Key characteristics of AI agents THINK 🧠 Analyze user’s prompt & data, system prompt, to define a goal to reach 🗺 PLAN Check available tools, define the strategy to realize the requested goal REFLECT ♻ Evaluate & loop over the output, to fix errors, to suggest improvements 🎬 ACT RAG searches, API calls, code execution, invoke other agents, request human’s help
7. ➡ Autonomous The agent decides on its own ➡ Prompt-driven The plan is described explicitly & given in the prompt ➡ External workflow An external program or workflow drives the LLMs Who’s planning? Who’s planning? Me or you?
8. ➡ Autonomous The agent decides on its own Prompt-driven The plan is described explicitly & given in the prompt External workflow An external program or workflow drives the LLMs Who’s planning? AGENT 🗺 LLM
9. Autonomous The agent decides on its own ➡ Prompt-driven The plan is described explicitly & given in the prompt External workflow An external program or workflow drives the LLMs Who’s planning? AGENT 🗺 LLM 🏼 💼
10. Autonomous The agent decides on its own Prompt-driven The plan is described explicitly & given in the prompt ➡ External workflow An external program or workflow drives the LLMs Who’s planning? AGENT LLM
11. ➡ Autonomous The agent decides on its own ➡ Prompt-driven The plan is described explicitly & given in the prompt ➡ External workflow An external program or workflow drives the LLMs Who’s planning? Function call hallucinations, wrong & unordered steps Deterministic, explicit & predictable, easier to maintain
12. ➡ Autonomous The agent decides on its own ➡ Prompt-driven The plan is described explicitly & given in the prompt ➡ External workflow An external program or workflow drives the LLMs Who’s planning? More autonomy, higher agency, adapted for changing env. Stricter plan, requires code maintenance for evolution
13. Agent design patterns 02
14. Function calling Chatbot app Gemini What’s the weather like in Paris? It’s sunny in Paris! External API or service user prompt + getWeather(String) function contract call getWeather(“Paris”) for me please 󰚦 getWeather(“Paris”) {“forecast”:”sunny”} function response is {“forecast”:”sunny”} Answer: “It’s sunny in Paris!”
15. Control flows https://huyenchip.com/2025/01/07/agents.html SEQUENTIAL TASK A TASK B PARALLEL TASK A TASK B CONDITIONAL ROUTING TASK A TASK B TASK C LOOPING TASK A
16. Why? Trust, safety, compliance, accountability, clarification, uncertainty, dilemmas… HITL — Human In The Loop Decisions have to be made, Human! Important decisions & actions should be made by a human being! ⚠
17. ReAct pattern (Reason / Act) LLM-as-Judge Ask an LLM to check the output, suggest improvements, fix errors Reflection & self-critique Thought Observe Act
18. Agentic RAG 03
19. Embedding model calculate RAG Vector DB vector embeddings chunks DOCS split store vector + chunk ❶ INGESTION
20. LLM context + prompt + chunks Embedding model calculate RAG Chatbot app Vector DB vector embeddings chunks DOCS prompt vector embedding find similar answer ❶ INGESTION ❷ RETRIEVAL
21. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. arXiv preprint arXiv:2210.01613 There are easy questions… and hard ones! Type Description Example Yes/No Answer is a Yes or No Has Lady Gaga ever made a song with Ariana Grande? Comparative Compare 2 items by an attribute Is Mont Blanc taller than Mount Rainier? Generic Simple questions Where was Michael Phelps born? Intersection Requires multiple conditions Which movie was directed by Denis Villeneuve and stars Timothee Chalamet? Ordinal Based on item's position in a list Who was the last Ptolemaic ruler of Egypt? Count Answer requires counting How many astronauts have been elected to Congress? Difference Contains a negation Which Mario Kart game did Yoshi not appear in? Superlative Max or Min of given attribute Who was the youngest tribute in the Hunger Games? Multi-hop Requires multiple steps to answer Who was the quarterback of the team that won Super Bowl 50?
22. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. arXiv preprint arXiv:2210.01613 There are easy questions… and hard ones! Type Description Example Yes/No Answer is a Yes or No Has Lady Gaga ever made a song with Ariana Grande? Comparative Compare 2 items by an attribute Is Mont Blanc taller than Mount Rainier? Generic Simple questions Where was Michael Phelps born? Intersection Requires multiple conditions Which movie was directed by Denis Villeneuve and stars Timothee Chalamet? Ordinal Based on item's position in a list Who was the last Ptolemaic ruler of Egypt? Count Answer requires counting How many astronauts have been elected to Congress? Difference Contains a negation Which Mario Kart game did Yoshi not appear in? Superlative Max or Min of given attribute Who was the youngest tribute in the Hunger Games? Multi-hop Requires multiple steps to answer Who was the quarterback of the team that won Super Bowl 50?
23. Agentic RAG Berlin’s origins, population, geographic situation 🧠 Agentic Assistant ————————————— 1) Identify topics 2) Create questions 3) RAG search 4) Collect answers & generate final report 🛠 History/Geography Tool ———————————————— 1) Execute RAG search 2) Call topic assistant to summarize topic 🧠 Topic Assistant ———————————— 1) Study topic answers 2) Create a report summary on the topic TOPICAL REPORTS FINAL REPORT Vector database TOPICAL REPORT
24. Sci-Fi story authoring agent 04 This is my story! To infinity & beyond!
25. https://short-ai-story.web.app/
26. Agent workflow 🧠 Story writer ————————— Write a story with a title, and 5 chapters. Write a story about {{type}} 🦜 LangChain4j ☕ ——————————— Drives the workflow via code Firestore database Final story 🧠 Image prompter ———————————— Create an image prompt about: {{chapter}} 🧠 Imagen —————————— Generate an image about: {{imgPrompt}} 🧠 Image judge ———————— Pick best {{images}} for {{chapter}} For each image & chapter… 🧠 Text enhance ———————— Make chapter more legible {{chapter}} For each chapter… 🎲 Story about time travel, nanobots, aliens encounter, cyberpunk… PING! chapter 4 images imgPrompt
27. 05 Model Context Protocol New protocol initiated…
28. MCP, the USB-C protocol for agent tools? https://norahsakal.com/blog/mcp-vs-api-model-context-protocol-explained/
29. MCP, the USB-C protocol for agent tools? https://norahsakal.com/blog/mcp-vs-api-model-context-protocol-explained/
30. Server MCP Host / Application Model Context Protocol MCP Server HTTP SSE MCP Server STDIO MCP Client MCP Client local resource local resource remote resource remote resource
31. Model Context Protocol — Initialization MCP Client MCP Server Initialize (Request session & capabilities) Initialize (Response w/ server capabilities) Notification/initialized (init. complete) Params: • protocolVersion • capabilities • clientInfo Result: • capabilities • serverInfo
32. Model Context Protocol — Tools MCP Client MCP Server tools/list (Request available tools) tools/list (Response w/ tools list) tools/call (Request tool execution) tools/call (Response w/ tool result) Params: • cursor (optional) Result: • tools • cursor (optional) Params: • name • arguments Result: • content • isError
33. Model Context Protocol — Resources MCP Client MCP Server resources/list (Request available resources) resources/list (Response w/ resources list) resources/read (Request specific resource content) resources/read (Response w/ specific resource) Result: • resources Params: • cursor (optional) Params: • url Result: • url • mimeType • text
34. Model Context Protocol — Prompts MCP Client MCP Server prompts/list (Request available prompts) prompts/get (Request specific prompt content) prompts/get (Response w/ specific prompt) Result: • prompts • cursor (optional) Params: • cursor (optional) Params: • name • arguments Result: • description • messages prompts/list (Response w/ prompts list)
35. Model Context Protocol — Notifications MCP Client MCP Server notifications/message (Send log message) Result: • level • logger • data
36. Model Context Protocol — Sampling MCP Client MCP Server sampling/createMessage (Request LLM sampling) sampling/createMessage (Response w/ sampling) Params: • message • modelPreferences • systemPrompt • maxTokens Result: • role • content • model • stopReason
37. The ‘S’ and the ‘O’ in MCP The ‘S’ in MCP stands for Security And the ‘O’ is for Observability
38. What happens in Vegas, doesn’t necessarily stay in Vegas…
39. 06 • ADK — Agent Development Kit • A2A — Agent to Agent protocol Hot off the press!
40. ADK — Google’s Agent Development Kit • New open source & code-first agent framework (already used internally at Google) • Supports Gemini, and any LLM via LiteLLM • Deployable anywhere ◦ your own server, cloud, etc. ◦ on Google Agent Engine ◦ containerized on Cloud Run
41. ADK — Google’s Agent Development Kit • Multi-agent: a hierarchy of agents • Flexible orchestration: sequential, parallel, loop • Session state management: short & long term • MCP support for tool calling & A2A for multi-agent scenarios
42. ADK — Google’s Agent Development Kit • Integrations with ◦ LLM frameworks: LangChain & LlamaIndex ◦ Agent frameworks: LangGraph & CrewAI • Bi-directional multimodal streaming • Built-in ◦ Command Line Interface ◦ UI web-based console ◦ Evaluation capabilities
43. ADK — Google’s Agent Development Kit Soon in Java! ☕ https://youtu.be/zgrOwow_uTQ?si=Vq4hUesBpPjTXnRx&t=237
44. A2A — Agent to Agent protocol Server — #A MCP Host / Application — #1 MCP Server HTTP SSE MCP Server STDIO MCP Client MCP Client Server — #B MCP Host / Application — #2 MCP Server HTTP SSE MCP Server STDIO MCP Client MCP Client A2A protocol • Agent discovery • Security & authentication • Task & state management • UX negotiation • Capability discovery
45. A2A — Agent to Agent protocol • An open standard to facilitate interoperability & collaboration between diverse AI agents, using different frameworks & platforms • Core architecture: client / server ◦ based on JSON-RPC 2.0 (like MCP) ◦ HTTP Server-Sent Events for streaming • Agent discovery: an agent card describe agents ◦ At a “well-known” location: /.well-known/agent.json ◦ Defines: skills, supported formats, endpoints, authentication
46. A2A — Agent Card { "name": "Google Maps Agent", "description": "Plan routes, remember places, and generate directions", "url": "https://maps-agent.google.com", "provider": { "organization": "Google", "url": "https://google.com" }, "version": "1.0.0", "authentication": { "schemes": "OAuth2" }, "defaultInputModes": ["text/plain"], "defaultOutputModes": ["text/plain", "application/html"], "capabilities": { "streaming": true, "pushNotifications": false }, ...
47. A2A — Agent Card ... "skills": [ { "id": "route-planner", "name": "Route planning", "description": "Helps plan routing between two locations", "tags": ["maps", "routing", "navigation"], "examples": [ "plan my route from Sunnyvale to Mountain View", "what's the commute time from Sunnyvale to San Francisco at 9AM", "create turn by turn directions from Sunnyvale to Mountain View" ], "outputModes": ["application/html", "video/mp4"] }, ... ] }
48. A2A — Core protocol components Task Central unit of work with a unique ID and lifecycle (submitted, working, completed…) Artifact Immutable outputs of a task, composed of one or more Parts Message Container for communication turns between client & agent, also composed of Parts Part Unit of content within Messages and Artifacts (TextPart, FilePart, DataPart) https://github.com/kweinmeister/agentic-trading/tree/main
49. Agent 2 Agent vs Model Context Protocol? A2A Standardize A2A communication Multi-agent workflow focus Discovery with cards Exchanges tasks & artifacts OAuth2 for authentication Coordination & delegation of autonomous agents MCP Standardize LLM / tool communication Enhance a single agent capability Tools, resources, prompts, sampling USB-C to plug tools Protocols for the agent ecosystem Aim for interoperability Can be used together to complement each other Still new and not widely adopted Security still a concern
50. Thanks for your attention (is all you need?) Guillaume Laforge Developer Advocate Ready for the AI agent future? glaforge glaforge.dev @glaforge @glaforge.dev @glaforge@uwyn.net
51. Illustrations courtesy of Imagen 3