Concerto for Java and AI - Building Production-Ready LLM Applications

Imagine you’re a music composer struggling to find inspiration for a pivotal movie scene. Then, you remember you’re also a software engineer, and the solution becomes suddenly obvious. Join me in this session, where I’ll demonstrate how I enhanced my music composition process by harnessing the power of Spring Boot and AI.

This presentation will discuss the core architectural patterns for introducing AI capabilities into an existing software system, exploring concepts like templated prompts, agent tools, and integration of external knowledge. The Java ecosystem is getting more and more capabilities for building AI applications. But are they ready for production? Are there any gaps?

Throughout the session, I’ll build a “composer assistant” application using Spring Boot and an AI Orchestrator (Spring AI) to showcase how to make an LLM application production-ready. Is the developer experience affected when working locally with language models? How is observability different when it comes to tokens? Can we ensure resilience across the many integrations orchestrated by the AI? What strategies are available for deploying LLM applications?

In a final twist, you’ll choose which movie scene to score, and I’ll compose and perform the music live for it, supported by AI. Will it meet the mark? There’s only one way to find out: join me in exploring the practical side of AI applications, where Java and Generative AI offer tangible solutions to real-world use cases. Aaaaand action!

Transcript

1. Thomas Vitale Spring I/O May 31st, 2024 Concerto for Java and AI Building Production-Ready LLM Applications @vitalethomas
2. Systematic • Software Engineer • CNCF Ambassador, Oracle ACE Pro, Testcontainers Community Champion • Author of “Cloud Native Spring in Action” (Manning). • OSS contributor (Java, Spring, Cloud Native Technologies) Thomas Vitale thomasvitale.com @vitalethomas
3. Generative AI @vitalethomas LLM RAG Prompting Embeddings Vector Stores Hallucinations
4. One Buzzword To Rule Them All @vitalethomas
5. The WHY Factor @vitalethomas
6. The WHY Factor @vitalethomas What problem does it solve? How ready is it for production? Yeah, but how about the DevEx?
7. Large Language Models @vitalethomas
8. AI != ML @vitalethomas
9. ML != LLM @vitalethomas
10. Machine Learning Subset of Arti fi cial Intelligence @vitalethomas Model Training Model Fine-Tuning Model Inference ML Engineers Platform/Infrastructure Platform Engineers HTTP API Application Developer
11. If you like it, you should put an API on it @vitalethomas
12. Model Inference via HTTP APIs Spring AI @vitalethomas Application Model Inference Service Do you wanna build a snowman? HTTP Spring Data Application Database Service DELETE * FROM HYPE; JDBC
13. Classification @vitalethomas
14. Classification @vitalethomas Application Model Service Classify HARMONY
15. LLM Security Risks (1) OWASP Top 10 for LLM Prompt Injection Model Denial of Service Sensitive Information Disclosure @vitalethomas OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/
16. Resilience LLM Applications @vitalethomas Resilience4J
17. Observability Distributed Tracing Request Rate Errors Duration Prompt Content Token Usage Context Window @vitalethomas
18. Semantic Search @vitalethomas
19. Semantic Search @vitalethomas From Keywords to Meaning Application Melancholic Embedding Model Melancholic [42…] LIKE ‘%melancholic%' Database Vector Store [42…]
20. Data Ingestion LLM Applications @vitalethomas Document Reader Document Transformer Document Writer JobRunr
21. Question Answering @vitalethomas
22. Question Answering with Docs @vitalethomas Retrieval Augmented Generation Application Melancholic instrument? Embedding Model Melancholic instrument? [42…] Get Similar Documents Vector Database Model Service Question + Similar Documents
23. One Does Not Simply Test LLM Applications @vitalethomas
24. Structured Data Extraction @vitalethomas
25. Structured Data Extraction @vitalethomas From Text to JSON Application Text Text to Structured JSON Model Service Database Save Structured JSON
26. LLM Security Risks (2) OWASP Top 10 for LLM Insecure Output Handling Excessive Agency Insecure Plugin Design @vitalethomas OWASP Top 10 LLM Applications and Generative AI https://genai.owasp.org/
27. Hallucinations @vitalethomas
28. RFBD @vitalethomas Randomly Failing By Design
29. Data Validation JSON Schema Humans in the Loop Optional Values @vitalethomas Mitigating hallucination risks
30. Speech Transcription @vitalethomas
31. Speech Transcription @vitalethomas From Speech to Text Application Audio Audio to Text Audio Model Chat Model Text to Structured JSON
32. Privacy @vitalethomas
33. Build & Deploy Cloud Native Buildpacks Kubernetes Service Binding Native Executables with GraalVM @vitalethomas Going to Production
34. Service Bindings for Spring AI @vitalethomas
35. Composer Assistant @vitalethomas https://github.com/ThomasVitale/concerto-for-java-and-ai
36. Thomas Vitale @vitalethomas thomasvitale.com Concerto for Java and AI Building Production-Ready LLM Applications