Java + LLMs: A Hands-on Guide with Bazlur Rahman & Syed M. "Shaaf"

Join us and learn how to build LLM apps with Java.

https://www.eventbrite.com/e/java-llms-a-hands-on-guide-with-bazlur-rahman-syed-m-shaaf-tickets-1309355159529

Transcript

1. Java + LLMs: A hands-on guide to building LLM Apps in Java Syed M Shaaf Sr. Principal Developer Advocate Red Hat Bazlur Rahman Java Champion 🏆 Staff Software Developer at DNAstack
2. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev 2 Java + LLMs: A hands-on guide to building LLM Apps in Java • Mentors, writes, speaks at conferences, and contributes to open-source projects outside of regular work hours. • Founder and current moderator of the Java User Group in Bangladesh (since 2013) • Recognized as the Most Valuable Blogger (MVP) at DZone, a leading technology publisher. • Serves as an editor for the Java Queue at InfoQ and at Foojay.io, a community for OpenJDK enthusiasts. • Authored five bestselling books about the Java programming language in Bengali. https://x.com/bazlur_rahman rokon12 https://www.linkedin.com/in/bazlur/ https://bazlur.ca/ https://bsky.app/profile/bazlur.ca
3. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Java + LLMs: A hands-on guide to building LLM Apps in Java • First line of Java in 1996. • Java developer, advocate, architect, engineer… • Contributor to Konveyor.io, Migrations of JavaEE using LLMs • #keycloak #infinispan #temurin #quarkus #kubernetes #ai/ml #fedora • InfoQ Java Technical Editor • Organizer Copenhagen Tech Talks (since 2013) • Mentor, writer and speaker at local meetups, events, conferences.. • Ask me about #Java, backends, architecture, opps, ML .. • (x) Cricket Coach - Boys and girls teams in Copenhagen, Denmark fosstodon.org/@shaaf sshaaf https://www.linkedin.com/in/shaaf/ shaaf.dev https://bsky.app/profile/shaaf.dev
4. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev • Systems do not speak Natural language, can’t translate and lack context outside of system boundaries. (e.g. sentiment) • Generating content is costly and sometimes hard. • Rapid data growth • Rising Expectations: Customers demand instant, personalized solutions. • Inefficiency: Manual processes increase costs and slow operations. • Skill Gaps: Limited expertise in AI adoption. Systems, Data, Networks and a Solution?
5. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Foundation models Learning without labels, adapt, tune, massive data appetite • Tasks ◦ Translation, Summarization, Writing, Q&A • “Attention is All you need”, Transformer architecture • Recognize, Predict, and Generate text • Trained on a Billions of words • Can also be tuned further A LLM predicts the next token based on its training data and statistical deduction Large Language Models
6. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Tokens Tokenization: breaking down text into tokens. e.g., Byte Pair Encoding (BPE) or WordPiece); handle diverse languages and manage vocabulary size efficiently. [12488, 6391, 4014, 316, 1001, 6602, 11, 889, 1236, 4128, 25, 3862, 181386, 364, 61064, 9862, 1299, 166700, 1340, 413, 12648, 1511, 1991, 20290, 15683, 290, 27899, 11643, 25, 93643, 248, 52622, 122, 279, 168191, 328, 9862, 22378, 2491, 2613, 316, 2454, 1273, 1340, 413, 73263, 4717, 25, 220, 7633, 19354, 29338, 15] https://platform.openai.com/tokenizer "Running", “unpredictability” (word-based tokenization). Or: "run" " ning" ; “un” “predict” “ability” (subword-based tokenization, used by many LLMs). “Building Large Language Models from scratch” - Sebastian Raschka
7. DEMO
8. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Basics - Basic htmx - Chat window - Backend sends question to the LLM. - Streaming is also an option
9. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Whats an AI Service? - AI Services, tailored for Java - similar to Spring Data JPA or Retrofit - handle the most common operations
10. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Prompts System prompt - Define the task - Set the expectations - Provide examples User prompt - Specific to the input When to use system vs user? What is a good prompt!? - E.g. Structure your input and output, (different LLMs behave differently) ** Try not to migrate prompts across models
11. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Chat and Memory • Eviction policy • Persistence • Special treatment of SystemMessage and Tools
12. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Function calling / Tools @Tool double squareRoot(double x) { return Math.sqrt(x); } - Call other services or functions to enhance the response. - E.g. Web APIs, internal system requests
13. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Amazing things Stupid mistakes… “..Do not mix accuracy with truth..”
14. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Retrieval Augmented Generation What is the representation of the data? How do I want to split? Per document Chapter Sentence How many tokens do I want to end up with? How much overlap is there between segments?
15. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev Tools + RAG
16. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev MCP - Model Context Protocol https://github.com/modelcontextprotocol - Standardized Context Format - Improved Interoperability - Richer Context Representation - Enhanced Model Grounding
17. https://github.com/learnj-ai/llm-jakarta @bazlur.ca @shaaf.dev - Demo use case - *My* Books API - Books MCP Server
18. 18 Thank you! Syed M Shaaf Developer Advocate Red Hat Bazlur Rahman Java Champion 🏆 Empowering Developers through Speaking 🗣 Writing ✍ Mentoring 🤝 & Community Building 🌍 Published Author 📖 Contributing Editor at InfoQ and Foojay.IO fosstodon.org/@shaaf sshaaf https://www.linkedin.com/in/shaaf/ shaaf.dev https://bsky.app/profile/shaaf.dev https://x.com/bazlur_rahman rokon12 https://www.linkedin.com/in/bazlur/ https://bazlur.ca/ https://bsky.app/profile/bazlur.ca Source for the demo https://github.com/learnj/llm-jakarta