ABCS25: It takes more than a prompt - implemeting GenAI for insurance by Gaetan Hervé & Yacine Ould Boukhitine

Implementing end to end use case leveraging GenAI under strict Finma and LPD regulation in Switzerland is a bit more complex than simply typing a prompt in ChatGPT. From data privacy, to proper orchestration including non regression, testing, validation, follow up, use case evolution and integration with external services, our speakers will guide you through their journey.
🙂 GAETAN HERVE ⚡️ Product Manager AI @ Vaudoise Assurance
🙂 YACINE OULD BOUKHITINE ⚡️ Cloud Technical Architect @ Swisscom

Transcript

1. It takes more than a prompt Implementing GenAI for insurance
2. Agenda • Introduction • Business Context • Architecture and application • Concerns & Solutions • Looking ahead
3. Introduction
4. Introduction https://huggingface.co/models
5. Speakers Gaétan Hervé • Vaudoise Assurance • Product Manager AI • EPFL – Microengineering • Passion : signal processing and clockwork • Out of work : Music, Football and Taxi for the kids ☺ Yacine Ould Boukhitine • Swisscom • Cloud Technical Architect • UCBL – System engineering • Passion : Shannon Information theory • Out of Work : Judo, Fishing in the Aare and Taxi for the kids as well ;)
6. Business Context
7. Vaudoise Assurance 7 Mobility Pet Health and Life Personal / Pro Liability Property Since 1895 La Vaudoise Assurance, one of the top private Swiss insurer provides insurance products for individuals, companies, public institution.
8. From the paper era to the PDF era
9. A new paradigm late 2022 Sure, below the information : This is Mr. James X. 12.03.25 is the beginning, 28.03.256 the end date. The doctor is Mr. Y. and the incapacity rate is 50% for accident Hi, can you extract me the name of the patient, the start date / end date of the sickness / accident and the rate of incapacity ?
10. Impact on Business Process Mr. X Du : dd/mm Au : dd/mm Taux : xx% Mr. X Du : dd/mm Au : dd/mm Taux : xx% 1 2 3 1 2 3 4 5
11. The reality Dev Ops Data Ops ML Ops LLM Ops <prompt> System Engineering AI / ML Ops 1 2 3 4 A B
12. In summary • This is all about system integration • One system asks a question • Your prepare the anwer • Another system digests the answer • (opt.) You produce valuable data System Calling System Processing AI (and more) New insights
13. Architecture
14. Deterministic/Non-Deterministic Bridging Transactional system • Require predictable outcomes for given inputs. • Often have strict time constraints for responses. • Operations are typically idempotent or have clear transactional boundaries. LLM's • Responses can vary even for the same input (due to temperature settings, model updates). • Response times can be unpredictable and sometimes lengthy. • "Creativity" and "contextual understanding" are strengths but contribute to non-determinism. How to reconcile these opposing characteristics without compromising either system's integrity?
15. Universal Enterprise Integrability (Machine 2 Machine) Our consumers are Vaudoise system/Application • For need the support of various communication protocols (AMQP, gRPC, HTTP) • We need to support different type of processing (Batch, Stream) • We are not alone :) we may need to provide consistent data contracts and full fill the demand !!! • We have to follow the technology grow and grow with it while keeping a standard with other systems
16. Agility & Evolvability Modular & Reusable • Each new use case shall not be a complete re design of the all system. We must be Agile (LLM's is moving fast) while keeping some simple to make it evolvable • DATA is AI :) -> Data engineer and Data scientist with different skills sets. The world of LLMs, new models, associated services, and integration patterns is evolving at an unprecedented pace.
17. The BUILD is important but remains the RUN at the end Have an eyes on any E2E transactions • Be able to understand who call, when for what. • OpenAI 429 Http code will appear - how to masterise them • Show Back and Charge Back ;)
18. And Many more …... Microsoft Well Architected Framework Cost Optimized, Secure, Reliable, Performance Efficient, Operationally Excellent Microsoft AI Gateway recommendation Content moderation, resources load balancing, ... O'REILLY Designing Data Intensive Application Dr. Werner Vogels The Frugal Architect
19. And now ?
20. Integrate, Orchestrate, Deliver, Observe IODO • Integrate: Seamless M2M with enterprise systems (diverse protocols, standard contracts). • Orchestrate: Agile, use-case driven GenAI workflows. •Deliver: Reliable, validated, compliant GenAI outcomes. •Observe: End-to-end monitoring (performance, cost, compliance, evolution).
21. Communication HUB Activities Activities Activities Activities Activities Orchestration Orchestration Orchestration Triggers API Triggers Service BUS Triggers Service BUS Various Azure Services Databricks Entreprise App/System
22. Which Azure service(s) host the Communication Hub ?
24. Integrates Azure function support 22x types of triggers – Thanks to the Azure function • We only expose 3 generic async API's request, status and feedbacks exposed on the APIm. • We load balance the request in our internal pool of plan and can autoscale based on events • We produce a result on a service bus Topic and use the built-in filtering system to redirect the message to the corresponding caller subscriptions
25. Orchestrate Orchestration Use case X Triggers API Triggers Service Bus Activity A Activity B Activity C Activity D • An orchestration is a statefull entities responsible for the management and ordering of the different activities. • An Activity is a stateless operation which based on an input provide an output (eg call databricks, call doc intelligence, store results) …..
26. Deliver reliable, validated and compliant outcomes. Foundation of Trust: As Code As Much as Possible • Infrastructure as Code (IaC) • Databricks Asset Bundle (dab) Rigorous Validation Path: • Continuous Integration with Unit and Integration, Code quality And package security scanning • System performance evaluation and testing (Load and Peformance) • Model None regression and accuracy • 6 Environnments to pass before reaching production
27. On the agent side Model Exposure ("Agent") Model Registry (MlFlow) Model Evaluation (MlFlow) Model Building (Langchain) Model Inference Reconciliation Feedbacks Asset Bundle Mosaic AI Delta (Live) Tables LLM End Points Mosaic AI Gateway
28. Observe
29. The Low Level Design
30. Concerns & Solutions
31. Repeatability ... • Need of multiple environments • Evolve without breaking oLLM Models -> as an API oYour "Agent" chain -> your own "docker" oYour data processing chain -> it's a data pipeline process • Terraform, Databricks Asset Bundle • You build once, you support for years ... As code, as code, ...
32. A new friend in the house ... • You don't own the model • Access, gateway and monitoring • A never ending race • Speed, Performance, Cost, Size • GPU GPU GPU • Deployment mode ... model management
33. Know the situation ... • Ideation : can we do it ? • Characterisation : how well can we do it ? • Drift monitoring : how long can we stand it ? ... system evaluation
34. Sleep well ... • Different layer oBusiness o"AI" aspect oSystem • Give retry a chance • Governance • => Need of multiple approaches ... Observability and respect
35. Looking Ahead
36. MCP • What it is • Why it's important • From … • AI won't replace you but someone with AI will • … TO • AI won't replace you but MCP will
37. An End to End process management • Call Center Management • Call to report an accident • First autonomous agent will check for completness of information • Second autonomous agent will check for coverage • Third autonomous agent will open the claim in the system • Next autonomous agents will ensure the completion based on information / payment treshold • BUT … • No denial without human intervention unless you want to meet the Swiss Law
38. In Short Integrate • Your GenAI alone won't provide any value • Plan early and ensure smooth integration (real agility) Orchestrate • You will likely need a feature more than once • Ensure you can orchestrate and reuse at best Deliver • The core of the Gen AI value, get a question provide an answer • Is your design ready for the specific aspect of Gen AI Observe • (Gen) AI can and will break (model, world changing, …) • Know before your user that changes are needed
39. Beyond the System to System • MCP at the core oMCPs your agent / tools oMake sure your agent can call your systems via MCP • Real world application => make your valuable endpoint accessible via Copilot through agent
40. You are not alone 10/06/2025 40 https://www.linkedin.com/in/gaetanherve/
41. Back to our friend https://huggingface.co/models ?
43. Q&A Time