Leverage LLMs in Java with LangChain4j and Quarkus

In this session, we’ll explore how to infuse AI capabilites into Java applications, using LangChain4j and its Quarkus integration. We’ll start from the Quarkus DevUI where you can try out AI models even before writing any code. Then we’ll explore LangChain4j features such as prompting, chaining, and preserving state; agents and function-calling; enriching your AI model’s knowledge with your own documents using retrieval augmented generation (RAG); and discovering ways to run (and train) models locally using tools like Ollama and/or Podman AI Lab. In addition, we’ll take a look at observability and fault tolerance of the AI integration and compile the app to a native binary.

Transcript

1. Georgios Andrianakis Principal Software Engineer, Quarkus Holly Cummins Senior Principal Software Engineer, Quarkus Leverage LLMs in Java with LangChain4j and Quarkus Open Conf November 22, 2024
2. The landscape
3. THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE © Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap) Version 1.0 - March 2024 Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email MAD2024@firstmarkcap.com AI MODELS AI FRAMEWORKS, TOOLS & LIBRARIES DATA & AI CONSULTING MLOPS & AI INFRA ESG LOCATION INTELLIGENCE DATA SOURCES & APIs AIR / SPACE / SEA FINANCIAL & MARKET DATA PEOPLE / ENTITIES OPEN SOURCE INFRASTRUCTURE QUERY / DATA FLOW STREAMING & MESSAGING STAT TOOLS & LANGUAGES COLLABORATION FORMATS DATA MANAGEMENT OLAP DATABASES SEARCH LOCAL AI VISUALIZATION LOGGING & MONITORING ORCHESTRATION PRIVACY & SECURITY FULLY MANAGED GRAPH DBs MPP DBs DATA GOVERNANCE & CATALOG COMPUTE GPU CLOUD / INFRA EDGE AI CLOSED SOURCE MODELS MGMT / MONITORING NewSQL DATABASES DATA INTEGRATION DATA WAREHOUSES DATA LAKES / LAKEHOUSES STREAMING / IN-MEMORY ORCHESTRATION REVERSE ETL REAL TIME DATABASES GPU DATABASES VECTOR DATABASES MULTI- MODEL DATABASES & ABSTRACTIONS APPLICATIONS — INDUSTRY APPLICATIONS — HORIZONTAL HUMAN CAPITAL DECISION & OPTIMIZATION MARKETING SALES CUSTOMER EXPERIENCE FINANCE & INSURANCE PARTNERSHIPS FINANCE AUTOMATION & OPERATIONS TEXT AUDIO & VOICE IMAGE PRESENTATION & DESIGN CODE & DOCUMENTATION LEGAL REGTECH & COMPLIANCE DATA SCIENCE NOTEBOOKS DATA SCIENCE PLATFORMS COMPUTER VISION SPEECH / VOICE NLP COMMERCIAL AI RESEARCH NONPROFIT AI RESEARCH ENTERPRISE ML/AI PLATFORMS AI OBSERVABILITY AI SAFETY & SECURITY DATA GENERATION & LABELING MLOPS AI DEVELOPER PLATFORMS AI HARDWARE AGRICULTURE HEALTHCARE INDUSTRIAL & LOGISTICS LIFE SCIENCES CROSS- INDUSTRY AEROSPACE, DEFENSE & GOV’T VIDEO EDITING SEARCH / CONVER- SATIONAL AI VIDEO GENERATION ANIMATION & 3D / GAMING TRANSPORTATION CUSTOMER DATA PLATFORMS LOG ANALYTICS ENTERPRISE SEARCH / KNOWLEDGE ANALYTICS BI PLATFORMS DATA ANALYST PLATFORMS PRODUCT ANALYTICS VISUALIZATION DATA MARKETPLACES & DISCOVERY DATA FRAMEWORKS NoSQL DATABASES ETL / ELT / DATA TRANSFORMATION RDBMS STORAGE DATA QUALITY & OBSERVABILITY INFRASTRUCTURE APPLICATIONS — ENTERPRISE MACHINE LEARNING & ARTIFICIAL INTELLIGENCE ANALYTICS INFRA- STRUCTURE AU LARGE 3 3 The landscape
4. THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE © Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap) Version 1.0 - March 2024 Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email MAD2024@firstmarkcap.com AI MODELS AI FRAMEWORKS, TOOLS & LIBRARIES DATA & AI CONSULTING MLOPS & AI INFRA ESG LOCATION INTELLIGENCE DATA SOURCES & APIs AIR / SPACE / SEA FINANCIAL & MARKET DATA PEOPLE / ENTITIES OPEN SOURCE INFRASTRUCTURE QUERY / DATA FLOW STREAMING & MESSAGING STAT TOOLS & LANGUAGES COLLABORATION FORMATS DATA MANAGEMENT OLAP DATABASES SEARCH LOCAL AI VISUALIZATION LOGGING & MONITORING ORCHESTRATION PRIVACY & SECURITY FULLY MANAGED GRAPH DBs MPP DBs DATA GOVERNANCE & CATALOG COMPUTE GPU CLOUD / INFRA EDGE AI CLOSED SOURCE MODELS MGMT / MONITORING NewSQL DATABASES DATA INTEGRATION DATA WAREHOUSES DATA LAKES / LAKEHOUSES STREAMING / IN-MEMORY ORCHESTRATION REVERSE ETL REAL TIME DATABASES GPU DATABASES VECTOR DATABASES MULTI- MODEL DATABASES & ABSTRACTIONS APPLICATIONS — INDUSTRY APPLICATIONS — HORIZONTAL HUMAN CAPITAL DECISION & OPTIMIZATION MARKETING SALES CUSTOMER EXPERIENCE FINANCE & INSURANCE PARTNERSHIPS FINANCE AUTOMATION & OPERATIONS TEXT AUDIO & VOICE IMAGE PRESENTATION & DESIGN CODE & DOCUMENTATION LEGAL REGTECH & COMPLIANCE DATA SCIENCE NOTEBOOKS DATA SCIENCE PLATFORMS COMPUTER VISION SPEECH / VOICE NLP COMMERCIAL AI RESEARCH NONPROFIT AI RESEARCH ENTERPRISE ML/AI PLATFORMS AI OBSERVABILITY AI SAFETY & SECURITY DATA GENERATION & LABELING MLOPS AI DEVELOPER PLATFORMS AI HARDWARE AGRICULTURE HEALTHCARE INDUSTRIAL & LOGISTICS LIFE SCIENCES CROSS- INDUSTRY AEROSPACE, DEFENSE & GOV’T VIDEO EDITING SEARCH / CONVER- SATIONAL AI VIDEO GENERATION ANIMATION & 3D / GAMING TRANSPORTATION CUSTOMER DATA PLATFORMS LOG ANALYTICS ENTERPRISE SEARCH / KNOWLEDGE ANALYTICS BI PLATFORMS DATA ANALYST PLATFORMS PRODUCT ANALYTICS VISUALIZATION DATA MARKETPLACES & DISCOVERY DATA FRAMEWORKS NoSQL DATABASES ETL / ELT / DATA TRANSFORMATION RDBMS STORAGE DATA QUALITY & OBSERVABILITY INFRASTRUCTURE APPLICATIONS — ENTERPRISE MACHINE LEARNING & ARTIFICIAL INTELLIGENCE ANALYTICS INFRA- STRUCTURE AU LARGE 3 3 The landscape 😵💫
5. THE 2024 MAD (MACHINE LEARNING, ARTIFICIAL INTELLIGENCE & DATA) LANDSCAPE © Matt Turck (@mattturck) , Aman Kabeer (@AmanKabeer11) & FirstMark (@firstmarkcap) Version 1.0 - March 2024 Blog post: mattturck.com/MAD2024 Interactive version: MAD.firstmarkcap.com Comments? Email MAD2024@firstmarkcap.com AI MODELS AI FRAMEWORKS, TOOLS & LIBRARIES DATA & AI CONSULTING MLOPS & AI INFRA ESG LOCATION INTELLIGENCE DATA SOURCES & APIs AIR / SPACE / SEA FINANCIAL & MARKET DATA PEOPLE / ENTITIES OPEN SOURCE INFRASTRUCTURE QUERY / DATA FLOW STREAMING & MESSAGING STAT TOOLS & LANGUAGES COLLABORATION FORMATS DATA MANAGEMENT OLAP DATABASES SEARCH LOCAL AI VISUALIZATION LOGGING & MONITORING ORCHESTRATION PRIVACY & SECURITY FULLY MANAGED GRAPH DBs MPP DBs DATA GOVERNANCE & CATALOG COMPUTE GPU CLOUD / INFRA EDGE AI CLOSED SOURCE MODELS MGMT / MONITORING NewSQL DATABASES DATA INTEGRATION DATA WAREHOUSES DATA LAKES / LAKEHOUSES STREAMING / IN-MEMORY ORCHESTRATION REVERSE ETL REAL TIME DATABASES GPU DATABASES VECTOR DATABASES MULTI- MODEL DATABASES & ABSTRACTIONS APPLICATIONS — INDUSTRY APPLICATIONS — HORIZONTAL HUMAN CAPITAL DECISION & OPTIMIZATION MARKETING SALES CUSTOMER EXPERIENCE FINANCE & INSURANCE PARTNERSHIPS FINANCE AUTOMATION & OPERATIONS TEXT AUDIO & VOICE IMAGE PRESENTATION & DESIGN CODE & DOCUMENTATION LEGAL REGTECH & COMPLIANCE DATA SCIENCE NOTEBOOKS DATA SCIENCE PLATFORMS COMPUTER VISION SPEECH / VOICE NLP COMMERCIAL AI RESEARCH NONPROFIT AI RESEARCH ENTERPRISE ML/AI PLATFORMS AI OBSERVABILITY AI SAFETY & SECURITY DATA GENERATION & LABELING MLOPS AI DEVELOPER PLATFORMS AI HARDWARE AGRICULTURE HEALTHCARE INDUSTRIAL & LOGISTICS LIFE SCIENCES CROSS- INDUSTRY AEROSPACE, DEFENSE & GOV’T VIDEO EDITING SEARCH / CONVER- SATIONAL AI VIDEO GENERATION ANIMATION & 3D / GAMING TRANSPORTATION CUSTOMER DATA PLATFORMS LOG ANALYTICS ENTERPRISE SEARCH / KNOWLEDGE ANALYTICS BI PLATFORMS DATA ANALYST PLATFORMS PRODUCT ANALYTICS VISUALIZATION DATA MARKETPLACES & DISCOVERY DATA FRAMEWORKS NoSQL DATABASES ETL / ELT / DATA TRANSFORMATION RDBMS STORAGE DATA QUALITY & OBSERVABILITY INFRASTRUCTURE APPLICATIONS — ENTERPRISE MACHINE LEARNING & ARTIFICIAL INTELLIGENCE ANALYTICS INFRA- STRUCTURE AU LARGE 3 3 The landscape 😵💫 😖
6. But I’m a Java developer. I do not want whitespace to have semantic significance.
7. Our goal: enabling developers to use AI models 4
8. A simplified landscape Left / Right of the Model
9. A simplified landscape Left / Right of the Model
10. A simplified landscape Left / Right of the Model
11. A simplified landscape Left / Right of the Model
12. A simplified landscape Left / Right of the Model
13. A simplified landscape Left / Right of the Model
14. A simplified landscape Left / Right of the Model
15. A simplified landscape Left / Right of the Model
16. A simplified landscape Left / Right of the Model
17. Langchain4j
18. Dependency <dependency> <groupId>io.quarkiverse.langchain4j</groupId> <artifactId>quarkus-langchain4j-openai</artifactId> <version>0.16.4</version> </dependency>
19. It all starts with prompts
20. Demo time 🎸 LangChain4j
21. @RegisterAiService interface Assistant { String chat(String message); } -------------------- @Inject private final Assistant assistant; quarkus.langchain4j.openai.api-key=sk-... Configure an API key Define Ai Service Use DI to instantiate Assistant
22. @SystemMessage("You are a professional poet") @UserMessage(""" Write a poem about {topic}. The poem should be {lines} lines long. """) String writeAPoem(String topic, int lines); Add context to the calls Main message to send Placeholder
23. Demo time 🎸 AIService API
24. Interaction flows need memory
25. @RegisterAiService(chatMemoryProviderSupplier = BeanChatMemoryProviderSupplier.class) interface AiServiceWithMemory { String chat(@UserMessage String msg); } --------------------------------- @Inject private AiServiceWithMemory ai; String userMessage1 = "Can you give a brief explanation of Kubernetes?"; String answer1 = ai.chat(userMessage1); String userMessage2 = "Can you give me a YAML example to deploy an app for this?"; String answer2 = ai.chat(userMessage2); Possibility to customize memory provider (Quarkus provides a default) Remember previous interactions
26. @RegisterAiService(/*chatMemoryProviderSupplier = BeanChatMemoryProviderSupplier.class*/) interface AiServiceWithMemory { String chat(@MemoryId Integer id, @UserMessage String msg); } --------------------------------- @Inject private AiServiceWithMemory ai; String answer1 = ai.chat(1,"I'm Frank"); String answer2 = ai.chat(2,"I'm Betty"); String answer3 = ai.chat(1,"Who Am I?"); default memory provider Refers to conversation with id == 1, ie. Frank keep track of multiple parallel conversations
27. Demo time 🎸 AIService API
28. Moving beyond text
29. Moving beyond text to objects
30. class TransactionInfo { @Description("full name") public String name; @Description("IBAN value") public String iban; @Description("Date of the transaction") public LocalDate transactionDate; @Description("Amount in dollars of the transaction") public double amount; } interface TransactionExtractor { @UserMessage("Extract information about a transaction from {{it}}") TransactionInfo extractTransaction(String text); } Marshalling objects
31. Demo time 🎸 AIService API
32. Moving beyond text to actions
33. Moving beyond text to logic
34. @RegisterAiService(tools = EmailService.class) public interface MyAiService { @SystemMessage("You are a professional poet") @UserMessage("Write a poem about {topic}. Then send this poem by email.") String writeAPoem(String topic); @ApplicationScoped public class EmailService { @Inject Mailer mailer; @Tool("send the given content by email") public void sendAnEmail(String content) { mailer.send(Mail.withText("letter@quarkus.io", "A poem", content)); } } Describe when to use the tool Register the tool Ties it back to the tool description
35. Demo time 🎸 AIService API
36. Thank you! red.ht/quarkus-langchain4j-tutorial http://hollycummins.com/llms-and-quarkus-openconf/