Say "Hi!" to Spring AI and LangChain4j - Seamless Support for GenAI in Your System

Generative AI is taking the world by storm. While some use cases may seem questionable, many systems can genuinely benefit from GenAI. While most attention still focuses on advancing GenAI capabilities, integration with new and existing enterprise-grade systems often gets overlooked. This talk presents a software architect's perspective on incorporating various GenAI models into your applications. We'll examine key quality attributes (such as portability, maintainability, security, and extensibility) while comparing two popular JVM solutions - Spring AI and LangChain4j. All of that will be illustrated with real-life stories and examples.

Transcript

1. Say "Hi!" to Spring AI and LangChain4j Piotr Ł a sk a wiec
2. „Describe me in 5 concise bullet points” Prompt: • Pragmatic Software Architect • Creating software for nearly 20 years • Founder of Comsenity • Helping companies to deliver real value • Occasional speaker and trainer Gener a ted by AI @piotrl a sk a wiec @piotrl a sk a wiec.bsky.soci a l w a ll.sli.do -> #geecon
3. *This chart is de fi nitely not scienti fi c… Hype over time Hype Time NoSQL Microservices Blockchain AI Tool/Language XYZ Event Horizon
4. Under 1 minute… GenAI/LLMs • Cre a tes new content - text, im a ges, code, a udio, video - b a sed on p a tterns le a rned from l a rge d a t a sets. • LLMs a re tr a ined speci f ic a lly on l a ngu a ge d a t a to underst a nd a nd gener a te hum a n- like text. • LLMs use billions of p a r a meters to predict a nd gener a te coherent, context- a w a re text outputs. • Tr a nsformer Architecture (token prediction). Arti f ici a l Inteligence Arti f ici a l Intelligence M a chine Le a rning Deep Le a rning Gener a tive AI Im a ge/Video LLM
5. ig.ft.com/gener a tive- a i/
6. poloclub.github.io/tr a nsformer-expl a iner/
8. Source: @Dexerto
9. Use cases • Summ a rizers • Assist a nts • Customer support • Cl a ssi f ic a tion • Customer sentiment a n a lysis • Converting unstructured to structured d a t a • Tr a nsl a tors
10. REST API Prompt Completion
11. Completion REST API Prompt Roles User/System Not st a nd a rdized St a teless Ch a t Memory Structured Output Pre-tr a ined System Inter a ctions Context D a t a V a rious Input Types V a rious Output Types Request/Response Stre a ming Fr a mework Integr a tion Observ a bility Test a bility
12. It shouldn’t be that hard, right?
13. LangChain4j
14. Common parts • Still not f in a l ;-) • Spring AI 1.0.0 - 20.05.2025 • L a ngCh a in4j 1.0.0 - M a y 2025 • Simil a r concepts a nd a bstr a ctions • Multiple LLM providers supported • Support for multimod a lity • Prompt templ a tes • Response stre a ming • Structured outputs • Ch a t memory • RAG (Retriev a l-Augmented-Gener a tion) • Multiple Vector Stores supported
15. • Deep integr a tion with Spring ecosystem • Out-of-the-box support for building MCP servers LangChain4j • Spring Boot, Qu a rkus, Helidon integr a tions • Kotlin extensions • Qu a rkus a dd-ons: • Tools execution model • E a syRAG • Dev UI integr a tion
16. Demo #1
17. Context is everything Prompt Stu ff ing RAG Tool C a lling Embeddings Vector Stores Ingestion
18. Tool Calling Applic a tion bound a ry Service A Service B Extern a l Model Extern a l Service D a t a source
19. Let LLM do the job Agentic AI • LLM is responsible for choosing how to solve a given problem using a v a il a ble tools. • There a re no gu a r a ntees reg a rding the control f low. • Tools c a n be c a lled 0-N times. • Ph a ntom tools (L a ngCh a in4j a llows to de f ine Tools H a llucin a tion Str a tegy).
20. Demo #2
21. Tool Calling Applic a tion bound a ry Service A Service B Extern a l Model Extern a l Service D a t a source
22. modelcontextprotocol.io
23. Model Context Protocol • C a p a bilities • Resources - generic text/bin a ry d a t a (d a t a b a se records, f ile content, API responses etc.) • Tools - execut a ble function a lities • Prompts - reus a ble templ a tes for users • Resources vs Tools • S a mpling • Tr a nsports • Stdio • HTTP/SSE • D a t a form a t - JSON-RPC 2.0
24. https://geecon.mcpserver.pl/sse
26. mcpm a rket.com
27. Interconnected Agents MCP MCP MCP
28. google.github.io/A2A/
29. Demo #3
30. Quality attributes • Security • Test a bility • Correctness • Perform a nce • Observ a bility
31. Security • Att a cks • Prompt Injection • Prompt Le a king • J a ilbre a king • DoS • Defense is in your h a nds • Gu a rdr a ils • Supported by L a ngCh a in4j Qu a rkus integr a tion
32. Demo #4
33. Testability • Simple checks • All required inform a tion provided in the a nswer • Proper tools were invoked • Token counts • How to test something th a t is inherently not deterministic? • M a ke it more deterministic • Fight f ire with f ire - AI Model Ev a lu a tion
34. Correctness • H a llucin a tions • Writing a good prompt is a n a rt • Provide qu a lity context to the model • Retries
35. Performance • Perform a nce-Oriented Prompt Engineering (sic!) • SLAs in non-deterministic world m a y be h a rd to re a ch • Lot of moving p a rts
36. Observability • Support for metrics a nd tr a ces • Follows OTel „Sem a ntic conventions for gener a tive AI systems” (still in development)
37. The end of explicit programming?
38. Natural language joins top programming languages
39. Thank you!