Turbocharging AI Innovation: Wie AI-Plattformen helfen GenAI Use Cases zuverlässig auf die Straße zu bringen @ CloudLand

Generative AI ist in aller Munde. Und wer auch nur 5 Minuten darüber nachdenkt, findet in seinem direkten Business Umfeld sicherlich sinnvolle Use Cases. Danach stehen wir allzu oft vor folgendem Dilemma: Wir wollen zügig raus in die Welt mit unseren Chatbots und Assistenzsystemen und unsere Ideen zur Marktreife bringen. Gleichzeitig bremsen wichtige, komplexe, querschnittliche Aspekte wie Datenschutz, Compliance, Betriebsreife oder das Feintuning der verwendeten Modelle eine schnelle Entwicklung und Bereitstellung häufig aus. Erschwerend kommt hinzu, dass bei der Umsetzung oft viele verschiedene Stakeholder involviert sind: Data Engineers, AI Specialists, Software Engineers, Betriebs-Experten und Fachabteilungen. Wir quatschen uns tot, statt unsere Use Cases gemeinsam schnell und effizient auf die Straße zu bringen. AI-Plattformen eilen zu Hilfe! Wir glauben: nur über etablierte Ansätze und Technologien des Platform Engineering, gepaart mit GenAI Ops Praktiken kommen wir raus aus dem Dilemma. Nur eine robuste, skalierbare und flexible Plattform ermöglicht es unseren Teams, ihre Daten, Modelle und Anwendungen effizient zu entwickeln, zu betreiben und zu verwalten. Die Plattform verbirgt die inhärente technische Komplexität, damit die Nutzer sich voll auf den Use Case und die Schaffung von Wert und Innovation konzentrieren können. Wir beleuchten, wie eine Corporate AI-Plattform aussehen kann und welche Bausteine und Services sie benötigt. Wir diskutieren, wie eine firmenweite Plattformstrategie nicht nur die technische Umsetzung vereinfacht, sondern auch ein Ökosystem für Innovation schafft, die Zusammenarbeit fördert, Wiederverwendbarkeit erhöht und letztendlich die Time to Market drastisch verkürzt.

Transcript

1. qaware.de Turbocharging AI Innovation How AI Platforms Enable The Bulletproof Deployment of GenAI Use Cases Mario-Leander Reimer © 2024 QAware
2. 2 Mario-Leander Reimer Managing Director | CTO @LeanderReimer #cloudnativenerd #qaware #gernperDude
3. The Next Frontier The Vast Universe of AI Use Cases
4. Welche KI Use Cases sind in euren Unternehmen geplant oder auch schon umgesetzt? ⓘ Click Present with Slido or install our Chrome extension to activate this poll while presenting.
5. Use Case: Customer Support Call Chat Support Resources Knowledge Database Support Flow Customer Data Images and Icons were generated with the assistance of AI
6. Use Case: Customer Support Support Resources Knowledge Database Support Flow Customer Data Images and Icons were generated with the assistance of AI RAG Automation Intent Recognition Text2Speech Speech2Text Anomaly Detection Similarity Matching AI Assistant Call Chat
7. Use Case: Content Generation Simple and easy generation of ▪ E-Mail responses ▪ Product videos ▪ Presentations ▪ User manuals ▪ Document summaries ▪ Tender documents ▪ Source code in the desired style. Images and Icons were generated with the assistance of AI
8. Use Case: Anomaly Detection, Predictive Maintenance, ... Image Source: https://azure.microsoft.com/en-us/products/ai-services/ai-anomaly-detector
9. The most common use cases for Gen AI Tools are marketing, sales, product development, and service operations. Source: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year
10. Wie funktioniert eigentlich ein Chatbot? Was steckt hinter RAG, Embeddings, .. ?
11. Wie funktioniert Retrieval Augmented Generation (RAG)? Index, z.B. Vector DB Indexing (Chunking & Embedding) Dokumente Ingestion Phase Query Encoding Retrieval Phase Context Prompt LLM mit Weltwissen Response
12. Vom Input zum Embedding. Das ermöglicht die performante semantische Suche mittels Vector Datenbanken. Embedding Model Images were generated with the assistance of AI { 23.567, 45.899, 76.345, …}
13. Wenn das nicht reicht, hilft Transfer Learning. Pre-trained Model Gut genug? Re-Training mit spezialisierten Daten Spezialisiertes Model Task ja nein Images were generated with the assistance of AI
14. Why do we need an AI platform?
15. But we are already doing this! … Really? MLOps only covers certain parts of the tasks around GenAI. Source: https://neptune.ai/blog/mlops https://dl.acm.org/doi/10.5555/2969442.2969519
16. Each involved stakeholder has a different expertise and therefore a different focus. These must be consolidated. Domain Expert Software Engineers and Architects This image was generated with the assistance of AI Data Scientists, AI Experts Platform Engineers
17. Models and talent pose the biggest challenges, alongside strategy as a frequent hindrance. Source: https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year
18. Your data and business domain are the driving force. Start here. CRISP-DM helps to start a structured AI approach. Business Understanding Data Understanding Data Preparation Modelling Evaluation Deployment Source: Shearer, C. (2000). The CRISP-DM model: The new blueprint for data mining. Journal of Data Warehousing, 5(4), 13-22
19. Chatbots and AI Assistants: Options and Development Stages ChatGPT or comparable with world knowhow ChatGPT with organisational context knowledge Specialized AI Assistent ▪ Retrieval Augment Generation ▪ Transfer Learning ▪ Custom trained modell ▪ Process Automation Complexity Benefit ▪ Easy to use and cost efficient ▪ Needs guidelines on data protection & compliance
20. “Too much cognitive load will become a bottleneck for fast flow and high productivity for many teams.” ▪ Intrinsic Cognitive Load Relates to fundamental aspects and knowledge in the problem space (e.g. languages, APIs, frameworks) ▪ Extraneous Cognitive Load Relates to the environment (e.g. console command, deployment, configuration) ▪ Germane Cognitive Load Relates to specific aspects of the business domain (aka. „value added“ thinking)
21. Platform engineering is the discipline of designing and building toolchains and workflows that enable self-service capabilities for software engineering organizations in the cloud-native era. Platform engineers provide an integrated product most often referred to as an “Internal Developer Platform” covering the operational necessities of the entire lifecycle of an application. https://platformengineering.org/blog/what-is-platform-engineering
22. A platform consists of different conceptual components. Depending on the stakeholders and their use cases. Developer Control Plane Integration and Delivery Plane Monitoring and Logging Plane Security Plane IDE Service Catalog / API Catalog Developer Portal Application Source Code Infrastructure & Platform Source Code Observability Secrets & Identity Manager CI Pipeline Registry CD Pipeline Resource Plane Compute Data Integration Networking Platform Orchestrator Certificates & Encryption GitOps https://humanitec.com/reference-architectures
23. Our proposal for an AI Platform Architecture
24. AI platform engineering is the discipline of designing and building toolchains and workflows to provide self-service capabilities for data and AI driven organizations. Business experts, data engineers as well as software engineers work together in an integrated platform from now on referred to as an “Enterprise AI Platform” covering the operational necessities of the entire lifecycle of AI use cases. © 2024, M.-Leander Reimer
25. Integration & Delivery Plane Service Plane Quality Plane Data Plane Platform Plane Observability Operability Resource Plane User Serving Plane Access Plane / APIs Orchestration Plane Data Modelling Plane Model Plane Compliance Plane Compute Data Integration Security Delivery FinOps
26. Quality Plane Integration & Delivery Plane Service Plane Access Plane/APIs User Serving Plane Technical and Business Metrics like Accuracy, Harmfulness, … Test Automation for LLMs „Convenience UIs“, Self Service, RAG per Drag and Drop, … (a) LLM, Embedding, (b) RAG, Chatbot, … (c) Data Access, … Orchestration Plane Data Modelling Pl. Playground Prompt Engineering Konfiguration Runtime, Instantiation, Orchestration, Scaling, Configuration Data Plane Ingestion Pipelines Data Versioning Embeddings & Vectorization Model Plane MLOps: Model Registry Model Management Experiment Tracking Model Serving Compliance Plane Tonality, Bias Security, Data Protection Platform Plane Observability: Monito- ring, Logging, Tracing Security: Secrets, IAM Encryption, Certs, … Scale, Backups, Recovery, … Delivery: CI/DC, Registry Pipelines, Orchestrator, … FinOps Resource Plane Compute: CPU and GPU Data: Vector DBasS, other Storage, … Integration: Self-hosted LLMs Public LLMs Managed AI Services
27. From concept to realisation: Possible variants
28. Many roads lead to Rome. Depending on your context, one or the other makes sense. Buy an AI platform solution + Convenience - Possibly not 100% suitable or extensible - Vendor Lock In Combination of cloud provider building blocks + Easily available - Vendor Lock In - Data Protection Considerations Custom platform with open source components + flexible and fully customizable - Time and Money for Setup and Maintenance
29. Azure AI Studio (Preview) Azure AI Content Safety Quality Plane Integration & Delivery Plane Service Plane Azure API Management Access Plane Azure AI Studio (Preview) User Serving Plane Azure AI Studio (Preview) Semantic Kernel Orchestration Plane Azure AI Document Intelligence Data Modelling Pl. Azure AI Search with Indexers, Indices incl. Vector DBs. OneLake, Fabric Data Plane Azure OpenAI Azure Machine Learning Model Plane Azure AI Content Safety Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data Azure OpenAI Azure AI Language Speech Service Azure AI Translator Integration Overview on Azure AI Services: https://learn.microsoft.com/en-us/azure/ai-services/what-are-ai-services
30. mlflow, Evidently AI, RAGAS (for RAG), DeepEval (for LLM) Quality Plane Integration & Delivery Plane Service Plane API Gateways Access Plane Build your own User Serving Plane Kubeflow Orchestration Plane Jupyter Kubeflow Data Modelling Pl. Weaviate, neo4J, … Custom Pipelines Data Plane mlflow (Registry) BentoML (Serving) Kubeflow (Serving) Model Plane Build your own Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data LLMs: Llama, Mistral, … mlflow BentoML Integration
32. mlflow, Evidently AI, RAGAS (for RAG), DeepEval (for LLM) Quality Plane Integration & Delivery Plane Service Plane API Gateways Access Plane Build your own User Serving Plane Kubeflow Orchestration Plane Jupyter Kubeflow Data Modelling Pl. Weaviate, neo4J, … Custom Pipelines Data Plane mlflow (Registry) BentoML (Serving) Kubeflow (Serving) Model Plane Build your own Compliance Plane Platform Plane Observability Security Scale, Backups, Recovery, … Delivery FinOps Resource Plane Compute Data LLMs: Llama, Mistral, … mlflow BentoML Integration Build at own risk or better ask an AI platform architect from QAware!
33. Welche Plattform Variante würdet ihr wählen? ⓘ Click Present with Slido or install our Chrome extension to activate this poll while presenting.
34. Which solution is the right one for me?
35. We suggest: Start lean and agile! Only with components required instead of "one size fits all"! Use Case Identification Business Understanding Skill, Resource & Requirements Analysis Building Block Mapping & Prioritization Implementation Evaluation Commoditization
36. QAware GmbH | Aschauer Straße 30 | 81549 München | GF: Dr. Josef Adersberger, Michael Stehnken, Michael Rohleder, Mario-Leander Reimer Niederlassungen in München, Mainz, Rosenheim, Darmstadt | +49 89 232315-0 | info@qaware.de Thank you!
37. We take responsibility and risks: From prototypes to large programs. We deliver. Guaranteed. 1. Our cross-functional teams of consultants, developers and managers see themselves as enablers. 2. We transform your organisation directly through project collaboration. With three guarantees: 3. Guarantee of success: We take responsibility and share your risks, for example through fixed prices. 4. Quality guarantee: You receive sustainable, reliable quality software – documented via KPIs and contractually fixed. 5. Satisfaction guarantee: We tie part of our remuneration to your satisfaction. 200 Engineers Munich Mainz Darmstadt Rosenheim Successful in the most demanding projects for 18 years Cloud Native Transformation & Host replacement: Tour guide into the future Data Value & AI: Open up data, network it & make it valuable From Allianz to Hellmann to Ericsson - we have been instrumental in their transformation. Pure impact: from prediction to AI assistants with valuable data and logic to the AI governance of tomorrow. When your boldest business ideas push the boundaries of IT, we push them together. 35 m € revenue Expertise for you Top provider: NPS 100 Top employer: 97% say "QAware is a very good workplace" Business Booster: Enable & accelerate business-critical visions Guaranteed success