Java + LLMs: A hands-on guide to building LLM Apps in Java with Jakarta

AI is revolutionizing the software landscape. However, for many Java developers, integrating these powerful AI tools into existing enterprise applications or a new one can feel daunting. In this hands-on session, we'll demystify the process and show you how to build LLM-powered features directly into your Java codebase.

Using JakartaEE and the LangChain4j library, we'll dive deep into Retrieval Augmented Generation (RAG), a cutting-edge technique that combines the vast knowledge of LLMs with the precision of your own data. We'll explore how to create both few-shot and zero-shot RAG models, and then add practical features like summarization and similarity search, backed by an Embedding database.

Through a live coding demo, we’ll walk you through constructing an AI-powered online store backend and provide practical insights into the architecture and code.

Whether you're familiar with AI or just getting started, this session will give you the confidence and skills to harness the potential of LLMs in your Java projects.

Source Code: https://github.com/learnj-ai/llm-jakarta

Transcript

1. 1 Java + LLMs: A hands-on guide to building LLM Apps in Java with Jakarta Syed M Shaaf Developer Advocate @ Red Hat Technical Editor @ InfoQ Bazlur Rahman Java Champion 🏆 Staff Software Developer at DNAstack
2. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Quick introduction • Prompts • Memory • Tools • RAG (Retrieval Augmented Generation) • Model Context Protocol What we will cover today
3. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Systems do not speak Natural language, can’t translate and lack context outside of system boundaries. (e.g. sentiment) • Generating content is costly and sometimes hard. • Rapid data growth • Rising Expectations: Customers demand instant, personalized solutions. • Inefficiency: Manual processes increase costs and slow operations. • Skill Gaps: Limited expertise in AI adoption. Systems, Data, Networks and a Solution?
4. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Understanding the journey that brought us here... Expert System Machine learning Deep learning Foundation models No use of data Manually authored rules Brittle Labour intensive Data prep, feature eng. Supervised learning, unsupervised learning, classification Learning without labels, adapt, tune, massive data appetite
5. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Foundation models Learning without labels, adapt, tune, massive data appetite • Tasks ◦ Translation, Summarization, Writing, Q&A • “Attention is All you need”, Transformer architecture • Recognize, Predict, and Generate text • Trained on a Billions of tokens • Can also be tuned further A LLM predicts the next token based on its training data and statistical deduction Large Language Models
6. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Tokens Tokenization: breaking down text into tokens. e.g., Byte Pair Encoding (BPE) or WordPiece); handle diverse languages and manage vocabulary size efficiently. [12488, 6391, 4014, 316, 1001, 6602, 11, 889, 1236, 4128, 25, 3862, 181386, 364, 61064, 9862, 1299, 166700, 1340, 413, 12648, 1511, 1991, 20290, 15683, 290, 27899, 11643, 25, 93643, 248, 52622, 122, 279, 168191, 328, 9862, 22378, 2491, 2613, 316, 2454, 1273, 1340, 413, 73263, 4717, 25, 220, 7633, 19354, 29338, 15] https://platform.openai.com/tokenizer "Running", “unpredictability” (word-based tokenization). Or: "run" " ning" ; “un” “predict” “ability” (subword-based tokenization, used by many LLMs). “Building Large Language Models from scratch” - Sebastian Raschka
7. Amazing things Stupid mistakes “..Do not mix accuracy with truth..” @bazlur.ca @shaaf.dev
8. Truth is Discrete not continuous @bazlur.ca @shaaf.dev
9. @bazlur.ca @shaaf.dev
10. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Langchain4J
11. DEMO
12. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev A Simple chat bot - Basic htmx - Chat window - Backend sends question to the LLM. - Streaming is also an option
13. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Whats an AI Service? - AI Services, tailored for Java - similar to Spring Data JPA or Retrofit - handle the most common operations
14. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Prompts System prompt - Define the task - Set the expectations - Provide examples User prompt - Specific to the input When to use system vs user? What is a good prompt!? - E.g. Structure your input and output, (different LLMs behave differently) ** Try not to migrate prompts across models
15. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Reasoning - Chain of Thought - TOT reasoning - Tree of Thought (Thinking, Organizing, Translating)
16. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Few-Shot , Zero Shot Zero-Shot - No data collection needed - Better accuracy with minimal examples - Lower accuracy on complex tasks Few-Shot - Fast implementation - Adaptable to niche tasks - Sensitive to example quality/order
17. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Chat and Memory • Eviction policy • Persistence • Special treatment of SystemMessage and Tools
18. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Function calling / Tools @Tool double squareRoot(double x) { return Math.sqrt(x); } - Call other services or functions to enhance the response. - E.g. Web APIs, internal system requests
19. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev MCP - Model Context Protocol - Standardized Context Format - Improved Interoperability - Richer Context Representation - Enhanced Model Grounding https://github.com/modelcontextprotocol
20. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev - Demo use case - *My* Books API - Books MCP Server
21. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev “feed relevant pieces of information (chunks) from your knowledge base to an LLM along with the user's query”... Retrieval Augmented Generation
22. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation What is the representation of the data? How do I want to split? Per document Chapter Sentence How many tokens do I want to end up with? How much overlap is there between segments?
23. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation Paragraph and Sentence splitters aim for semantic coherence but have variable chunk sizes. Character (especially recursive) and Word splitters offer size control but risk breaking semantic meaning. Line splitters are for specific line-oriented formats. Regex splitters provide maximum flexibility for known structures.
24. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Quick introduction • Prompts • Memory • Tools • RAG (Retrieval Augmented Generation) • Model Context Protocol What we covered….
25. 2 5 Thank you! Syed M Shaaf Developer Advocate Red Hat Bazlur Rahman Java Champion 🏆 Empowering Developers through Speaking 🗣 Writing ✍ Mentoring 🤝 & Community Building 🌍 Published Author 📖 Contributing Editor at InfoQ and Foojay.IO fosstodon.org/@shaaf sshaaf https://www.linkedin.com/in/shaaf/ shaaf.dev https://bsky.app/profile/shaaf.dev https://x.com/bazlur_rahman rokon12 https://www.linkedin.com/in/bazlur/ https://bazlur.ca/ https://bsky.app/profile/bazlur.ca Source for the demo https://github.com/learnj-ai/llm-jakarta https://docs.langchain4j.dev/ LangChain4J