Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Java and AI: Building Production-Ready LLM Appl...

Java and AI: Building Production-Ready LLM Applications (Without the Hype)

You’re a software engineer, and your project has to integrate AI. Hopefully, not just any AI. You probably need solutions that are private, efficient, and production-ready, not just a checkbox for the latest trend. Join me in this session, where we’ll apply The WHY Factor to cut through the hype and focus on what actually works.

In this hands-on session, we’ll explore how to build robust LLM applications using Java, open-source tools, and European machine learning models, ensuring compliance, security, and developer-friendly workflows. You’ll learn how to:

- Leverage open-source and European LLMs for inference, reducing dependency risks and keeping your data under your control.
- Design modular RAG architectures with Docling, enabling private, resource-efficient, and hallucination-free document processing.
- Integrate agentic workflows securely, connecting your enterprise applications and APIs without compromising privacy.
- Monitor, test, and refine your Generative AI applications locally, using observability and evaluation strategies for real-world reliability.

We’ll build a live demo to show how Java and Spring AI can help you integrate AI responsibly: local-first, open-source, and hype-free. By the end of this session, we’ll have a working application, and a bigger question: Does this AI solution actually address a need, or is it just another trend we’re chasing?

Avatar for Thomas Vitale

Thomas Vitale

March 12, 2026
Tweet

More Decks by Thomas Vitale

Other Decks in Technology

Transcript

  1. Thomas Vitale Tech Hub Aarhus Day 12th Mar 2026 Java

    and AI Building Production-Ready LLM Applications (Without the Hype) @thomasvitale.com
  2. The WHY Factor What problem does it solve? How ready

    is it for production? You get a great dev experience? @thomasvitale.com
  3. Machine Learning Subset of Arti fi cial Intelligence Platform/Infrastructure Platform

    Engineers HTTP API Application Developer Model Training Model Inference ML Engineers Data Preparation Data Scientists @thomasvitale.com
  4. Model Inference via HTTP APIs Application Model Inference Service Same

    procedure as last year? HTTP Same procedure as every year Application Database Service @thomasvitale.com DELETE * FROM HYPE; JDBC 42
  5. Java Application Architecture Observability Platform Exports telem etry Inference Service

    Consum es LLM s @thomasvitale.com Database Reads/writes data Reads/writes data Document Service Processes docs Spring Boot Application Arconia Spring AI
  6. Arconia Dev Services Zero-code, zero-con fi g external services >

    arconia dev > gradle bootRun > mvn spring-boot:run @thomasvitale.com https://docs.arconia.io
  7. Guardrails Inference Service Request Response Input and Output Input Output

    Application Input Guardrail Output Guardrail @thomasvitale.com
  8. Retrieval Augmented Generation Inference Service Request Response Question Answer Application

    Augment with Context Prompt Augmentation with Retrieved Context Source Query @thomasvitale.com
  9. Docling Java HTTP Integration Docling Java Java Application HTTP Service

    Docling Serve HTTP https://github.com/docling-project/docling-java @thomasvitale.com
  10. Retrieval Augmented Generation Inference Service Request Response Question Answer Application

    Augment with Context Vector Stores Vector Store Semantic Search @thomasvitale.com
  11. Chat Memory Inference Service Request Response Question Answer Application Multiple

    Interactions @thomasvitale.com Augment with Memory Memory Read Update Memory Write
  12. Tools Inference Service Request Tool Calling Question Response Answer Application

    API Tool Call Tool Execution Tool Call Request Tool Call Response @thomasvitale.com
  13. API MCP Server MCP Inference Service Request Tools Question Response

    Answer Application Tool Call MCP Client Tool Call Request Tool Call Response @thomasvitale.com
  14. Thomas Vitale @thomasvitale.com thomasvitale.com Java and AI Building Production-Ready LLM

    Applications (Without the Hype) https://github.com/ThomasVitale/tech-hub-aarhus-day-2026 https://github.com/ThomasVitale/modular-rag