From Zero to Production: Build Your Own GenAI Solution.

Thanks to powerful frameworks and libraries, the first Generative AI applications can be realized at Hello World level with just a few lines of code. However, these first attempts also reach their limits just as quickly. Why? Because reality presents challenges that cannot be solved easily with this trivial approach. But what is needed for a Generative AI application at enterprise level? A well designed Generative AI architecture!
During this full day hands-on workshop, we will build a complex Generative AI application step by step. Starting with a minimalist RAG system, we will look at various challenges and discuss suitable solutions. In the end, we will have an architecture that can withstand the challenges of reality.

Transcript

1. Generative AI From Zero to Production: Build your own GenAI solution #WISSENTEILEN powered by Lars Röwekamp | Tim Wüllner | open knowledge GmbH Hands-on Workshop
2. (Architecture, Cloud, AI & ML) Lars Röwekamp LinkedIn: lars-roewekamp CIO New Technologies OPEN KNOWLEDGE
3. Tim Wüllner Machine Learning Engineer OPEN KNOWLEDGE
4. What‘s GenAI? How does „Generative AI“ actually work?
6. The Idea: The question and answer are SEMANTICALY very similar. How does an LLM generates a response?
7. The Idea: The question and answer are SEMANTICALY very similar. How does an LLM generates a response?
8. The Idea: The question and answer are SEMANTICALY very similar. Sematic proximity is represented MATHEMATICALLY with the aid of vectors. How does an LLM generates a response?
9. Dog Cat Tiger Mouse Hamburg Berlin Madrid Amazon Embedding Model 0.6 0.3 0.1 … 0.8 0.5 0.3 … 0.4 0.2 0.9 … Embeddings (semantic vectors)
10. Embedding Model 0.6 0.3 0.1 … 0.8 0.5 0.3 … 0.4 0.2 0.9 … Berlin: City, Europe, Capital of Germany, … Dog Cat Tiger Mouse Hamburg Berlin Madrid Amazon Embeddings (semantic vectors)
11. Dog Cat Mouse Tiger Hamburg Berlin Madrid Amazon Embedding Model 0.6 0.3 0.1 … 0.8 0.5 0.3 … 0.4 0.2 0.9 … Berlin: City, Europe, Capital of Germany, … Dog Cat Tiger Mouse Hamburg Berlin Madrid Amazon
12. Dog Cat Mouse Tiger Hamburg Berlin Madrid Amazon Embedding Model 0.6 0.3 0.1 … 0.8 0.5 0.3 … 0.4 0.2 0.9 … Dog Cat Tiger Mouse Hamburg Berlin Madrid Amazon
13. Dog Cat Mouse Tiger Hamburg Berlin Madrid Amazon Embedding Model 0.6 0.3 0.1 … 0.8 0.5 0.3 … 0.4 0.2 0.9 … Dog Cat Tiger Mouse Hamburg Berlin Madrid Amazon
14. Tiger Hamburg Berlin Madrid Amazon „Can you name a few European capitals?“ Embedding Model 0.8 0.5 0.4 … QUESTION: „Can you name ..?“: City, Europe, Capital of … Dog Cat Mouse
15. Tiger Hamburg Berlin Madrid Amazon „Can you name a few European capitals?“ Embedding Model 0.8 0.5 0.4 … QUESTION: „Can you name ..?“: City, Europe, Capital of … Dog Cat Mouse
16. Tiger Hamburg Berlin Madrid Amazon „Can you name a few European capitals?“ Embedding Model 0.8 0.5 0.4 … QUESTION: „Can you name ..?“: City, Europe, Capital of … Our goal? Dog Cat Mouse
17. Hamburg Berlin Madrid Amazon Our goal? Do everything possible to ensure that the ‚question‘ and the ‚answer‘ ars as close together as possible!
18. Source: http://vectors.nlpl.eu/explore/embeddings/en/MOD_enwiki_upos_skipgram_300_2_2021/cat_NOUN/
19. man woman king queen Okay, but what‘s so great about that?
20. man woman king queen queen – woman + man = king Okay, but what‘s so great about that? We can calculate with it and ‚understand‘.
21. man woman king queen doctor – man + woman = ? Okay, but what‘s so great about that? We can calculate with it and ‚understand‘.
22. man woman king queen doctor – man + woman = nurse Okay, but what‘s so great about that? We can calculate with it and ‚understand‘.
23. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? word2vec style vectors
24. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? John wants his bank to cash the … ? (verb) (verb) context aka hidden state
25. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? John wants his bank to cash the … ? (verb) (verb) context aka hidden state Key-Vector: btw I‘am a noun describing a male person. Query-Vector: btw I‘am seeking for a noun describing a male person.
26. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? Transformer (LLM Layer) context aka hidden state John wants his bank to cash the … ? (verb) (verb) (John‘s) (finance) John wants his bank to cash the … ? (verb) (verb) (male) ( ... )
27. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? John (main character, male, married to Cheryl, cousin of Donald, from Minnesota, currently in Boise, … ) wants his bank to cash the … ?
28. Generative AI under the hood Transformer (LLM Layer) John wants his bank to cash the … ? Transformer (LLM Layer) John wants his bank to cash the … (verb) (verb) (John‘s) (finance) John wants his bank to cash the … ? (verb) (verb) (male) Enough context information to be able to „guess“ the next word. cheque ( ... )
29. Generative AI under the Hood Large Language Model John wants his bank to … Large Language Model Next Token increase cash close … Probability 0.2 0.4 0.1 … Next Token account cheque money order … Probability 0.15 0.75 0.05 … John wants his bank to cash the …
30. GenAI by Example? Where can I realistically use GenAI - and where not?
31. Generative AI by Example
32. Health Care & Pharmaceuticals Advertisment & Marketing Media & Entertainment Production & Manufactoring Financial Services Software Development Generative AI am Beispiel
33. Generative AI by Example Health Care Enhancing medical images Discovering new drugs* Simplifying medical tasks Persionalized treatment *via Generative Design
34. GenAI by Example „Know Your Business!“ *if you want to succeed in the end.
35. GenAI Basics How to build a first, simple GenAI system?
36. GenAI- Model Prompt GenAI Basics „What is the most beautiful holiday destination?“
37. Model A-small Model Provider model-name model-parameter provider-parameter prompt Model A-big Model B e.g. openAI, Google, … Provider Client API GenAI Basics Model Integration „What is the most beautiful holiday destination?“ Reminder: 00-hello-genai.ipynb
38. Hands-On Create a Jupyter Notebook in the Google Colab environment. Calling a GenAI model with a prompt via a provider-specific API. Hello IPYNB*, hello GenAI *Python Notebook aka Jupyter Notebook
39. https://colab.research.google.com/
40. Hands-on Hello Colab https://colab.research.google.com/
41. https://github.com/openknowledge/workshop-genai-student
42. Hands-on Hello GenAI World Jupyter Notebook https://github.com/openknowledge/workshop-genai-student
43. Hands-on Hello GenAI World Jupyter Notebook https://github.com/openknowledge/workshop-genai-student
44. GenAI- Model Prompt GenAI Basics „What is the most beautiful holiday destination?“
45. GenAI- Model Prompt GenAI Basics Model Selection Prompt Engineering 2 1 „What is the most beautiful holiday destination?“
46. GenAI explained MODELS
47. GenAI- Model Prompt GenAI Basics Model Selection Prompt Engineering 2 Model Selection „What is the most beautiful holiday destination?“ Welches Model? Welche Parameter? Welche Infrastruktur? Welche Lizenz? Welches Preismodell? Welche Governance? 1
48. GenAI Basics Model Selection „In the world of GenAI, model selection is part of the product design. It affects user experience, cost, scalability, and reliability.“ „While we might not pick the perfect model on the first try, starting with clear priorities like speed, cost, accuracy helps guide the process.“
49. Step 1 Step 2 Step 3 Step 4 GenAI Basics Model Selection aka „How do I find* the right model?“
50. Define specific GenAI use cases Clearly outline the purpose and application of GenAl. 1. Use Case Step 2 Step 3 Step 4 GenAI Basics Model Selection aka „How do I find* the right model?“
51. Define specific GenAI use cases Create a shortlist of foundation models Models that meet the business and technical requirements. 1. Use Case 2. Shortlist Step 3 Step 4 GenAI Basics Model Selection aka „How do I find* the right model?“ Clearly outline the purpose and application of GenAl.
52. Define specific GenAI use cases Create a shortlist of foundation models Evaluate and test selected models Models that meet the business and technical requirements. Conduct thorough testing to evaluate model performance in your own context. 1. Use Case 2. Shortlist 3. Evaluation Step 4 GenAI Basics Model Selection aka „How do I find* the right model?“ Clearly outline the purpose and application of GenAl.
53. Define specific GenAI use cases Create a shortlist of foundation models Evaluate and test selected models Integrate the best models and evaluate them on an ongoing basis Models that meet the business and technical requirements. Conduct thorough testing to evaluate model performance in your own context. Integrate, regularly evaluate and, if necessary, optimise the performance of the model.. 1. Use Case 2. Shortlist 3. Evaluation 4. Integration GenAI Basics Model Selection aka „How do I find* the right model?“ Clearly outline the purpose and application of GenAl.
54. GenAI Basics Model Selection aka „How do I find* the right model?“ „Know Your Use-Case!“ *wenn du am Ende erfolgreich sein möchtest.
55. GenAI- Model Prompt GenAI Basics Model Integration Prompt Engineering 2 Model Selection 1 Suitable model found: But how do I address it from my application? „What is the most beautiful holiday destination?“
56. Model A-small Model Provider model-name model-parameter provider-parameter prompt Model A-big Model B e.g. openAI, Google, … Provider Client API GenAI Basics Model Integration „What is the most beautiful holiday destination?“ Reminder: 00-hello-genai.ipynb
57. Model X mini Model Provider II model-name model-parameter provider-parameter prompt Model Y Model Y ++ e.g. openAI, Google, … Provider Client API GenAI Basics Model Integration „What is the most beautiful holiday destination?“
58. Model X mini Model Provider II Provider II Client API model-name model-parameter provider-parameter prompt Model Y Model Y ++ e.g. openAI, Google, … GenAI Basics Model Integration „What is the most beautiful holiday destination?“ Die openAI API ist aktuell der de-facto Standard.
59. Hub-Provider Client API model-name model-parameter hub-provider-parameter prompt e.g. Hugging-Face Provider-II hosted Models Provider-I hosted Models Hub-hosted Models* Hub-Provider Dispatcher *Provider X, fine-tuned or adapted, own GenAI Basics Model Integration „What is the most beautiful holiday destination?“ Cloud based Resources
60. Hub-Provider Client API model-name model-parameter hub-provider-parameter prompt e.g. Hugging-Face Provider-II hosted Models Provider-I hosted Models Hub-hosted Models* Hub-Provider Dispatcher *Provider X, fine-tuned or adapted, own GenAI Basics Model Integration „What is the most beautiful holiday destination?“ cloud-not-allowed exception
61. Local-Hub Provider Client API model-name model-parameter hub-provider-parameter prompt e.g. GPT4all, ollama Local hosted Models (Provider III) Local hosted Models (Provider II) Local hosted Models (own or adaptedI) Local-Hub Provider Dispatcher Local Machine GenAI Basics Model Integration „What is the most beautiful holiday destination?“
62. GenAI Basics Model Adaption „ What additional levers do I have?“ • Temperature degree of ‘fantasy‘ • Max Tokens length of answer • Top K selection of hits from the top K hits • Top P selection of hits from the top P per cent • Presence Penalty avoid repetition • Frequence Penalty avoid repetition (weighted edition)
63. GenAI explained PROMPTING
64. GenAI- Model Prompt Model Selection Prompt Engineering 2 1 GenAI Basics Prompt Engineering „What is the most beautiful holiday destination?“
65. GenAI- Model Prompt Model Selection Prompt Engineering 1 Parts of a Prompt System vs User Prompt Principles of Prompting Patterns for Prompting Context Window Query vs Chat GenAI Basics Prompt Engineering 2 „What is the most beautiful holiday destination?“
66. GenAI Basics Prompt Engineering Role Instruction Example 1 Context Example n Question Who am I? What is my intention? What are helpful examples? Are there any additional information? BTW: what is the task I ask for? What should a good prompt should look like?
67. GenAI Basics Prompt Engineering Role Instruction Example 1 Context Example n Question What is your role as an assistant? What is my intention? What are helpful examples? Are there any additional information? BTW: what is the task I ask for? What should a good prompt should look like?
68. GenAI Basics Good Prompt vs. Bad Prompt You I want to cook something. LR Topic: Recipe Recommendations
69. GenAI Basics Good Prompt vs. Bad Prompt You I want to cook something. LR You Acting as an expert home cook, for someone who enjoys vegetarian Italian food and has only 30 minutes to prepare dinner, could you recommend a recipe including a list of ingredients and step-by- step instructions? LR Output Example Context Question Role Acting as an expert home cook, for someone who enjoys vegetarian Italian food and has only 30 minutes to prepare dinner, could you recommend a recipe including a list of ingredients and step-by-step instructions? You could point at recipes you like from the BBC’s Good Food guide, providing URL’s to recipes you love. Topic: Recipe Recommendations
70. GenAI Basics Good Prompt vs. Bad Prompt You I want to learn something new. LR You Acting as a coding instructor, for a beginner with a goal to learn Python within 4 weeks, please provide a learning plan including resources and a weekly schedule for 10 hours per week. LR Output Example Context Question Role Acting as a coding instructor, for a beginner with a goal to learn Python within 4 weeks, Please provide a learning plan, including resources and a weekly schedule for 10 hours per week Point it at courses you’ve done in the past that you’ve liked. Tell it you like to learn by reading books, or websites, or by watching videos, or a mixture of both. Topic: Learning new skills
71. GenAI Basics System Prompt vs. User Prompt System Assistant User „You“ Chat-Model Behaviour of Assistant
72. GenAI Basics System Prompt vs. User Prompt System Assistant User ‘You're a friendly assistant.’ „You“ Chat-Model Behaviour of Assistant
73. GenAI Basics System Prompt vs. User Prompt System Assistant User ‘You're a friendly assistant and always keep things brief.’ ‚Which is the most beautiful city in the world?‘ ‚Which is the most beautiful city in the world?‘ ‘You are a philosopher and have a tendency to be verbose.’ ??? ??? „You“ Chat-Model Behaviour of Assistant
74. GenAI Basics System Prompt vs. User Prompt System-Prompt: ‘You're a friendly assistant and always keep things brief.’ Assistant: 65 Tokens
75. GenAI Basics System Prompt vs. User Prompt System-Prompt: ‘You are a philosopher and have a tendency to be verbose.’ Assistant: 434 Tokens
76. GenAI Basics System Prompt vs. User Prompt System-Prompt: Default GPT-4o Assistant: 296 Tokens
77. GenAI Basics Patterns for Prompting Pros & Cons Consideration Stepwise Interaction Flipped Interaction Discussion of Experts I‘am planing to travel to China. I‘am planing a trip to China. I‘am planing a trip to China. I‘am planing a trip to China.
78. GenAI Basics Patterns for Prompting I‘am planing to travel to China. Please analyze the pros and cons of traveling by bike, ship or plane. Pros & Cons Consideration Stepwise Interaction Flipped Interaction Discussion of Experts I‘am planing a trip to China. I‘am planing a trip to China. I‘am planing a trip to China.
79. GenAI Basics Patterns for Prompting I‘am planing to travel to China. Please analyze the pros and cons of traveling by bike, ship or plane. Pros & Cons Consideration Stepwise Interaction Flipped Interaction Discussion of Experts I‘am planing a trip to China. Walk me through the planing process step by step, explaining your reasoning at each stage. Wait for my con- firmation before moving to the next step. I‘am planing a trip to China. I‘am planing a trip to China.
80. GenAI Basics Patterns for Prompting I‘am planing to travel to China. Please analyze the pros and cons of traveling by bike, ship or plane. Pros & Cons Consideration Stepwise Interaction Flipped Interaction Discussion of Experts I‘am planing a trip to China. Walk me through the planing process step by step, explaining your reasoning at each stage. Wait for my con- firmation before moving to the next step. I‘am planing a trip to China. Before providing a solution, please ask me relevant questions about the trip, so that you can give me the most appropriate planing advice.“ I‘am planing a trip to China.
81. GenAI Basics Patterns for Prompting I‘am planing to travel to China. Please analyze the pros and cons of traveling by bike, ship or plane. Pros & Cons Consideration Stepwise Interaction Flipped Interaction Discussion of Experts I‘am planing a trip to China. Walk me through the planing process step by step, explaining your reasoning at each stage. Wait for my con- firmation before moving to the next step. I‘am planing a trip to China. Before providing a solution, please ask me relevant questions about the trip, so that you can give me the most appropriate planing advice.“ I‘am planing a trip to China. I am torn between sustainable and cheap travelling. Can you provide two schools or trends within the travel community that can represent the two approaches.
82. GenAI Basics Prompt Injection https://gandalf.lakera.ai/
83. Hands-On Prompting with a system Prompting using various prompting patterns and best practices. Extended prompting via chatbot using a storybook and chat history. Prompting & Models
84. Hands-on Prompting & Models
86. GenAI explained myDOMAIN
87. GenAI- Model Prompt GenAI Basics myDomain „What is the most beautiful holiday destination?“
88. GenAI- Model Prompt GenAI Basics myDomain „How to apply for vacation at open knowledge?“
89. GenAI- Model Prompt GenAI Basics myDomain + „Leave regulation at open knowledge“ „How to apply for vacation at open knowledge?“
90. GenAI- Model Prompt GenAI Basics myDomain + „Leave regulation at open knowledge“ „How to activate the alarm system at open knowledge?“
91. GenAI- Model Prompt GenAI Basics myDomain + + „How to activate the alarm system at open knowledge?“ „Leave regulation + Alarm system usage at open knowledge“
92. GenAI- Model Prompt GenAI Basics myDomain + „All internal knowledge (Wiki, DB, …) at open knowledge“ „How to activate the alarm system at open knowledge?“
93. GenAI- Model Prompt GenAI Basics myDomain + ERROR: Token Limit exceeded Output Size Context Window Size „How to activate the alarm system at open knowledge?“ „All internal knowledge at open knowledge“
94. (Source: https://www.meibel.ai/post/understanding-the-impact-of-increasing-llm-context-windows)
95. GenAI- Model Prompt GenAI Basics myDomain + ERROR: Out-of-Budget Exception Context Window Size „All internal knowledge at open knowledge“ „How to activate the alarm system at open knowledge?“
96. GenAI- Model Prompt GenAI Basics myDomain + ERROR: Lost-in-the-Middle Context Window Size „All internal knowledge at open knowledge“ „How to activate the alarm system at open knowledge?“
97. GenAI- Model Prompt GenAI Basics myDomain + ERROR: Self-fullfilling Prophecy Context Window Size Context Window Size Context Window Size Context Window Size „Alles interne Wissen (Wiki, DB, …) bei open knowledge“ „How to activate the alarm system at open knowledge?“
98. GenAI Advanced How can I contribute and query my own domain knowledge?
99. GenAI- Model Prompt GenAI Advanced myDomain + ERROR: Token Limit exceeded! Output Size Context Window Size „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
100. myGenAI Model Prompt GenAI Advanced myDomain „All internal knowledge at open knowledge“ „How to apply for vacation at open knowledge?“
101. myGenAI Model Prompt GenAI Advanced myDomain Option 1: Build own Model Option 2: Fine-tune existing Model ERROR: Way to expensive ERROR: Way to complex WARNING: Out-of-Sync „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
102. Prompt GenAI Advanced myDomain „Prompt Enrichment“ „Model Enrichment“ myGenAI Model + „How to apply for vacation at open knowledge?“
103. Prompt GenAI Advanced myDomain GenAI- Model SOME MAGIC „MAGIC Enrichment“ „How to apply for vacation at open knowledge?“
104. Prompt GenAI- Model GenAI Advanced myDomain SOME MAGIC „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
105. Prompt Retrieve Augment Ingesting- Pipeline Knowledge Database Retriever GenAI- Model GenAI Advanced myDomain „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
106. „How to apply for vacation at open knowledge?“ Prompt Ingesting- Pipeline Knowledge Database Retriever GenAI Advanced myDomain R etrieval Augmented Generation „All internal knowledge at open knowledge“
107. GenAI explained RAG Systems
108. Prompt Ingesting- Pipeline Knowledge Database Retriever GenAI Advanced RAG Systems R etrieval Augmented Generation „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
109. Prompt Ingesting- Pipeline Knowledge Database GenAI Advanced RAG Systems R etrieval Augmented Generation Retriever „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
110. GenAI Advanced RAG Systems GenAI Model INGESTING Pipeline (Async.) Chunking Embedding RETRIEVAL Pipeline (Sync.) Top-k Fetching Augmenting UI/UX (Prompt) Query- Embedding „How to apply for vacation at open knowledge?“ Retriever
111. GenAI Advanced RAG Systems GenAI Model INGESTING Pipeline (Async.) Chunking Embedding RETRIEVAL Pipeline (Sync.) Top-k Fetching Augmenting UI/UX Query- Embedding It looks complicated, and it is! Except in ‘Hello World’ examples from the Internet. Retriever
112. Chunking Embedding Ingesting Pipeline by Example Document (Doc. Id:1) „It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype. The added value of this technology is too great and its use in the form of a chat is too intuitive. In 2024, it was expected that the leap from exploratory playing around with large language models [...].“
113. Chunking Embedding Ingesting Pipeline by Example [„ It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype.“] [„ The added value of this technology is too great and its use in the form of a chat is too intuitive.“] Document (Doc. Id:1) „It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype. The added value of this technology is too great and its use in the form of a chat is too intuitive. In 2024, it was expected that the leap from exploratory playing around with large language models [...].“ [ … ]
114. Chunking Embedding Ingesting Pipeline by Example [-0.24, 0.15, …, 0.52] [-0.13, 0.95, …, 0.31] […] [„ It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype.“] [„ The added value of this technology is too great and its use in the form of a chat is too intuitive.“] Document (Doc. Id:1) „It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype. The added value of this technology is too great and its use in the form of a chat is too intuitive. In 2024, it was expected that the leap from exploratory playing around with large language models [...].“ [ … ]
115. Chunking Embedding Ingesting Pipeline by Example Id Doc. Id Embedding 1 1 [-0.24, 0.15, …, 0.52] 2 1 [-0.13, 0.95, …, 0.31] […] […] […] [-0.24, 0.15, …, 0.52] [-0.13, 0.95, …, 0.31] […] [„ It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype.“] [„ The added value of this technology is too great and its use in the form of a chat is too intuitive.“] Document (Doc. Id:1) „It will soon be the second anniversary of the release of ChatGPT. There is still no end in sight to the hype. The added value of this technology is too great and its use in the form of a chat is too intuitive. In 2024, it was expected that the leap from exploratory playing around with large language models [...].“ [ … ]
116. Retrieval Pipeline by Example Embedding Top-k Fetching Augmenting ‘What is the lecture ‘The architecture for language models in practice - Retrieval Augmented Generation’ about?’
117. Retrieval Pipeline by Example Embedding Top-k Fetching Augmenting [0.35, 0.25, …, 0.62] ‘What is the lecture ‘The architecture for language models in practice - Retrieval Augmented Generation’ about?’
118. Retrieval Pipeline by Example Embedding Top-k Fetching Augmenting Embeddings from knowledge base Embeddings Dimension 1 Embed. Dim. 2 Top-k=1 document(s) [0.35, 0.25, …, 0.62] ‘What is the lecture ‘The architecture for language models in practice - Retrieval Augmented Generation’ about?’
119. Retrieval Pipeline by Example Embedding Top-k Fetching Augmenting Embeddings from knowledge base Embeddings Dimension 1 Embed. Dim. 2 Top-k=1 document(s) [0.35, 0.25, …, 0.62] & „ What is the lecture […] based on the following context: ‘An upstream retrieval process to pull in relevant information from our own dynamic database can provide well-founded answers.’ ‘What is the lecture ‘The architecture for language models in practice - Retrieval Augmented Generation’ about?’
120. Sherlock RAG in Action
124. Hands-On Implementing a simple RAG system using ingestion and retrieval pipelines. Applying the RAG system to questions from your own domain. RAG System Simple Version
125. Hands-on Simple RAG
126. Retrieval Augmented Knowledge Database Retriever Generation Ingesting- Pipeline GenAI Advanced RAG Systems „How to apply for vacation at open knowledge?“
127. Retrieval Augmented Knowledge Database Retriever Generation Ingesting- Pipeline GenAI Advanced RAG Systems „How to apply for vacation at open knowledge?“ „Working like this you doesn't need a holiday. Lazy piece!“
128. GenAI Professional What is still missing for a sustainable and productive operation?
129. Retrieval Augmented Knowledge Database Retriever Generation Ingesting- Pipeline GenAI Professional Guardrails „How to apply for vacation at open knowledge?“ „Working like this you doesn't need a holiday. Lazy piece!“
130. Retrieval Augmented Knowledge Database Retriever Generation Guarding Output Ingesting- Pipeline GenAI Professional Guardrails Output Profanity Check „How to apply for vacation at open knowledge?“ „Working like this you doesn't need a holiday. Lazy piece!“
131. Retrieval Augmented Knowledge Database Retriever Generation Guarding Output Ingesting- Pipeline GenAI Professional Guardrails „Unfortunately, I can't answer that!“ Output Profanity Check „…. Lazy piece! …“ „How to apply for vacation at open knowledge?“ „Working like this you doesn't need a holiday. Lazy piece!“
132. Retrieval Augmented Knowledge Database Retriever Generation Input Output Ingesting- Pipeline GenAI Professional Guardrails Guarding Topic Check „Which Netflix series do you recommend?“
133. Retrieval Augmented Knowledge Database Retriever Generation Input Output Ingesting- Pipeline GenAI Professional Guardrails Guarding Topic Check „Which Netflix series do you recommend?“ „Sorry, but this does not fall within my area of responsibility.“
134. Retrieval Augmented Knowledge Database Retriever Generation Input Output Ingesting- Pipeline GenAI Professional Guardrails Output-Governance: Toxic Speech, Fact-Checking, Ethical-Guidelines, Cross-Validation, … Input-Evaluation: Prompt-Injection, Jailbreak-Attemp, Topic-Filter, PII-Cleansing, Rate-Limiting,… Guarding
135. Sherlock RAG in Action
137. Hands-On Create your own guardrails for input and output to prevent toxic language. Implementing a simple Output Guardrail as a fact checker. RAG System Guardrails
138. Hands-on Guardrails to the rescue
139. Hands-on Guardrails to the rescue
140. Retrieval Augmented Knowledge Database Retriever Generation Input Output Ingesting- Pipeline GenAI Professional Guarding What else is missing for a professional and productive system?
141. Retrieval Augmented Knowledge Database Retriever Generation Input Output Ingesting- Pipeline RAG-Monitoring Cost- Monitoring Model- Monitoring Response-Quality- Monitoring GenAI Professional Monitoring Guarding Ragas: https://docs.ragas.io/en/stable/concepts/metrics
142. Retrieval Augmented Generation relevanter Kontext generierte Antwort User Input GenAI Professional Monitoring
143. Generated Answer Retrieved Context User Input ‚Is the answer supported by the context?‘ ‚Is the context found relevant to the question?‘ ‚Is the answer relevant to the question?‘ GenAI Professional Monitoring: The RAG-Triad
144. Generated Answer Retrieved Context User Input ‚Is the given answer similar to the correct answer?‘ ‚Has the entire context to be found been found?‘ Ground Truth Answer Ground Truth Context ‚Is the answer supported by the context?‘ ‚Is the context found relevant to the question?‘ ‚Is the answer relevant to the question?‘ GenAI Professional Monitoring: The RAG-Triad+
145. Retrieval Augmented Generation GenAI Professional Response Quality Monitoring https://docs.confident-ai.com/docs/metrics-llm-evals G-Eval Deep Acyclic Graph Answer Relevancy Faithfulness Summerization Hallucination Contextual Recall Contextual Relevancy Contextual Precision Relevant Context Generated Answer User Input
146. Retrieval Augmented Generation User Input GenAI Professional Response Quality Monitoring Faithfulness Is the supported by the ? Relevant Context Generated Answer
147. Retrieval Augmented Generation User Input GenAI Professional Response Quality Monitoring Does the match the ? Relevant Context Generated Answer Answer Relevancy
148. GenAI Professional Response Quality Monitoring Step 1: Generate possible questions ( ) from Embeddings Dimension 1 Embed. Dim. 2 Answer Relevancy Does the match the ?
149. GenAI Professional Response Quality Monitoring Step 1: Generate possible questions ( ) from Step 2: Determine the average distance from Embeddings Dimension 1 Embed. Dim. 2 Answer Relevancy Does the match the ?
150. RAG in Action aka Sherlock RAG
152. Best Practices What are the typical pitfalls that I will run into - and how do I deal with them?
153. Retrieval Augmented Knowledge Database Retriever Generation Guarding Input Output Ingesting- Pipeline RAG-Monitoring Cost- Monitoring Model- Monitoring Response-Quality- Monitoring GenAI Best Practices Real Life Survival Guide
154. ‘The answers sound conclusive, but they are wrong!’ GenAI Best Practices Real Life Survival Guide
155. GenAI Model UI/UX Guardrails Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Query- Embedding Top-k Fetching Augmenting Input Hallucination (via Faithfulness / Answer Relevancy) GenAI Best Practices Real Life Survival Guide
156. GenAI Model UI/UX Guardrails Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Query- Embedding Top-k Fetching Augmenting Input Missing Data GenAI Best Practices Real Life Survival Guide
157. GenAI Model UI/UX Guardrails Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Query- Embedding Top-k Fetching Augmenting Input Missing Data GenAI Best Practices Real Life Survival Guide Chunk not found
158. GenAI Model UI/UX Guardrails Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Query- Embedding Top-k Fetching Augmenting Input Missing Data GenAI Best Practices Real Life Survival Guide Chunk not found Chunk not relevant
159. Ingesting-Pipeline (Async.) Chunking Embedding Real Life RAG #1 Missing-Data Challenge Missing Data
160. Ingesting-Pipeline (Async.) Chunking Embedding Real Life RAG #1 Missing-Data Challenge Missing Data Problem: RAG does not know the required data.
161. Ingesting-Pipeline (Async.) Chunking Embedding Real Life RAG #1 Missing-Data Challenge Missing Data Problem: RAG does not know the required data. Solution : Implement a suitable ingestor.
162. Ingesting-Pipeline (Async.) Chunking Embedding Real Life RAG #1 Missing-Data Challenge Missing Data Problem: Snapshot of the data is not up-to-date.
163. Ingesting-Pipeline (Async.) Chunking Embedding Real Life RAG #1 Missing-Data Challenge Missing Data Problem: Snapshot of the data is not up-to-date. Solution: Interface for ‘continuous’ data sync.
164. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found
165. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found.
166. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found. Solution #1: Adapt chunk-sizes and overlapping.
167. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found. Solution #2: Choose a different splitter (e.g. Semantic-Splitter).
168. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found. Solution #3: Optimize the distance threshold or k from top-k.
169. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found. Solution #4: Selection of a more suitable embedding model.
170. Query- Embedding Top-k Fetching Ingesting-Pipeline (Async.) Chunking Embedding Augmenting Retrieval-Pipeline (Sync.) Real Life RAG #2 Chunk-not-found Challenge Chunk not found Problem: Existing chunks will not be found. Solution #5: Hypothetical document embeddings (HyDE) Input HyDE
171. Guardrails Input Output Gao et al. (2023) HyDE = Hypothetical Document Embeddings Real Life RAG #2 Chunk-not-found Challenge HyDE
172. Real Life RAG #3 Chunk-not-relevant Challenge Chunk not relevant GenAI Model Query- Embedding Top-k Fetching Augmenting Retrieval-Pipeline (Sync.)
173. Real Life RAG #3 Chunk-not-relevant Challenge Problem: Chunk is not considered to be relevant by the model. Chunk not relevant GenAI Model Query- Embedding Top-k Fetching Augmenting Retrieval-Pipeline (Sync.)
174. Real Life RAG #3 Chunk-not-relevant Challenge Chunk not relevant Query- Embedding Top-k Fetching Augmenting Retrieval-Pipeline (Sync.) Problem: Chunk is not considered to be relevant by the model. Solution #1: Select a more suitable model for the use case. GenAI Model
175. Real Life RAG #3 Chunk-not-relevant Challenge Chunk not relevant Query- Embedding Top-k Fetching Augmenting Retrieval-Pipeline (Sync.) GenAI Model Problem: Chunk is not considered to be relevant by the model. Solution #2: Add additional context to the chunk (small-to-big). Chunk
176. GenAI Model UI/UX Guardrails Input Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Query- Embedding Top-k Fetching Augmenting
177. Hands-On Surviving the RAG jungle: Using LLM and RAG metrics to optimise the retrieval process. RAG System typical Pitfalls
178. Hands-on Surviving the RAG jungle
179. Generated Answer Retrieved Context User Input ‚Is the given answer similar to the correct answer?‘ ‚Has the entire context to be found been found?‘ Ground Truth Answer Ground Truth Context ‚Is the answer supported by the context?‘ ‚Is the context found relevant to the question?‘ ‚Is the answer relevant to the question?‘ GenAI Professional Monitoring: The RAG-Triad+
180. 1 Provide evaluation data that includes the ground truth context. 2 Generate responses to evaluation input and collect context. Real Life RAG Metrics in Information Retrieval
181. 1 2 Generate responses to evaluation input and collect context. 3 Determine whether context is included in ground truth. 1 0 Hit Rate Real Life RAG Metrics in Information Retrieval – Hit Rate Provide evaluation data that includes the ground truth context.
182. 1 2 Generate responses to evaluation input and collect context based on relevance. 3 Determine the proportion of relevant documents in relation to all documents. 𝑃𝑟𝑒𝑐𝑖𝑠𝑖𝑜𝑛 = 𝑟𝑒𝑙𝑒𝑣𝑎𝑛𝑡 𝑅𝑒𝑠𝑢𝑙𝑡𝑠 𝑡𝑜𝑡𝑎𝑙 𝑅𝑒𝑠𝑢𝑙𝑡𝑠 0.00 0.33 Real Life RAG Metrics in Information Retrieval – Precision Provide evaluation data that includes the ground truth context.
183. 1 2 Generate responses to evaluation input and collect context based on relevance. … 3 Determine the rank of the first relevant document. Calculate and average the reciprocal rank (RR): 𝑅𝑅 = 1 𝑅𝑎𝑛𝑔 1 2 3 1 2 3 1 2 0 𝑀𝑅𝑅 = 1 𝑛 σ𝑖=1 𝑛 𝑅𝑅𝑖 𝑀𝑅𝑅 = 0.25 1 2 3 1 2 3 Real Life RAG Metrics in Information Retrieval – Mean Reciprocal Rank (MRR) Provide evaluation data that includes the ground truth context.
184. Outlook What else is possible in addition to the pure knowledge query?
185. Prompt Retrieve Augment Ingesting- Pipeline Knowledge Database Retriever GenAI- Model GenAI Outlook What else is there to consider? „All internal knowledge at open knowledge“ „How to apply for vacation at open knowledge?“
186. GenAI explained I18N
187. UI/UX Enterprise Integration „Comment demander des vacances chez ok?“ Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model GenAI Outlook Multi-Language RAG Knowledge Database „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
188. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model Language- Detection Translation Translation Translation GenAI Outlook Multi-Language RAG Knowledge Database „All internal knowledge at open knowledge“ „Comment demander des vacances chez ok?“
189. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Knowledge Database Retriever GenAI- Model Translation Translation Translation Translation FR_fr language not supported domain specific wording GenAI Outlook Multi-Language RAG Language- Detection „Comment demander des vacances chez ok?“ „All internal knowledge at open knowledge in multiple languages.“
190. GenAI explained ACCESS CONTROL
191. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model Knowledge Database GenAI Outlook Access Control „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
192. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model Knowledge Database GenAI Outlook Access Control „What are the top secret KPIs of open knowledge?“ „All internal knowledge at open knowledge“ incl C-Level information.“
193. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model Token Knowledge Database Input Guardrail Some kind of Attack? Prompt-Injection? Prompt-Abuse? Rate-Limit? GenAI Outlook Access Control „What are the top secret KPIs of open knowledge?“ „All internal knowledge at open knowledge“ incl C-Level information.“
194. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Knowledge Database with AC Meta Data Retriever GenAI- Model RBAC via PBF Token Token Input Guardrail GenAI Outlook Access Control RBAC = Role based Access Control PBF = Permission based Filtering „What are the top secret KPIs of open knowledge?“ „All internal knowledge at open knowledge“ incl C-Level information.“
195. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Knowledge Database with AC Meta Data Retriever GenAI- Model RBAC via PBF Token Token Input Guardrail Output Guardrail PII: Personally identifiable information? GenAI Outlook Access Control „What does the personal data of X look like at ok?“ „All internal knowledge at open knowledge“ incl C-Level information.“
196. GenAI explained INTEGRATION
197. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Retriever GenAI- Model Knowledge Database GenAI Outlook Integration „All internal knowledge at open knowledge“ „How to apply for vacation at open knowledge?“
198. GenAI Outlook Integration GenAI based Service A P I ML as a Service Input Output UI/UX Enterprise Integration „All internal knowledge at open knowledge“ „How to apply for vacation at open knowledge?“
199. Hands-On Provision of a GenAI-based RAG system as a service. Integration of the service into a proprietary web-based interface. RAG System GenAI as a Service
200. Hands-on GenAI-as-a-Service
201. Hands-on GenAI-as-a-Service
202. GenAI explained AGENTS
203. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Knowledge Database Retriever GenAI- Model GenAI Outlook Multi-Agenten Systeme „How to apply for vacation at open knowledge?“ „All internal knowledge at open knowledge“
204. UI/UX Enterprise Integration Retrieve Augment Ingesting- Pipeline Knowledge Database Retriever GenAI- Model GenAI Outlook Multi-Agenten Systeme „All internal knowledge at open knowledge“ ‚Apply for my vacation from […] to [… ] at ok, please.‘
205. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX Agentic AI ‚Apply for my vacation from […] to [… ] at ok, please.‘
206. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX ‚You are an assistant for employees. Perform the following steps to complete the tasks: [...]‘ Agentic AI ‚Apply for my vacation from […] to [… ] at ok, please.‘
207. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX I should first retrieve past interactions with the user and user information. Agentic AI ‚You are an assistant for employees. Perform the following steps to complete the tasks: [...]‘ ‚Apply for my vacation from […] to [… ] at ok, please.‘
208. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX User : {Name: „Tim Wüllner“, ID: „123“} […] Agentic AI ‚Apply for my vacation from […] to [… ] at ok, please.‘
209. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX User : {Name: „Tim Wüllner“, ID: „123“} […] I should now obtain information on how to apply for leave at open knowledge. Agentic AI ‚Apply for my vacation from […] to [… ] at ok, please.‘
210. Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX ‚You can apply for holiday leave at open knowledge via the BCS system. Leave is applied for as follows […]. Agentic AI ‚Apply for my vacation from […] to [… ] at ok, please.‘
211. Agentic AI Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX I should now use the proposed tool(s) to apply for the holiday for the user as described. ‚You can apply for holiday leave at open knowledge via the BCS system. Leave is applied for as follows […]. ‚Apply for my vacation from […] to [… ] at ok, please.‘
212. Agentic AI Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX requestVacation(userId=„123“,from=„x“,to=„y“) ‚Apply for my vacation from […] to [… ] at ok, please.‘
213. Agentic AI Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX requestVacation(userId=„123“,from=„x“,to=„y“) I should now inform the user about the holiday request. ‚Apply for my vacation from […] to [… ] at ok, please.‘
214. Agentic AI Retriever Memory Tools Planning Database Prompting GenAI- Model API-Call UI/UX ‚Apply for my vacation from […] to [… ] at ok, please.‘ „I applied for … .“
215. UI/UX Spezialist für Urlaub Spezialist für User- Interaktion Spezialist für Coding Spezialist für Social Media ‚Create a proposal for a contribution on the topic of AI agents.‘ ‚Why does this code result in a NullPointerException ?‘ Agentic AI Multi Agent Systems
216. Conclusion What should I take away from the workshop for myself and my team?
217. GenAI Model UI/UX Guardrails Input Output Ingesting-Pipeline (Async.) Chunking Embedding Retrieval-Pipeline (Sync.) Top-k Fetching Augmenting Query- Embedding
218. GenAI from prototype to production Your personal „takeaways“: • GenAI is powerful. But also expensive. • Professional prompting is your super power. • Every model has its own character. • RAG for your own domain knowledge. • GenAI is also just software. Your use case determines the right path!
219. Time for Questions? Yes, sure!
220. #WISSENTEILEN Many Thanks! #WISSENTEILEN by open knowledge GmbH @_openKnowledge | @mobileLarson Lars Röwekamp, CIO New Technologies
221. Looking beyond the RAG horizon: Cognitive Architecture blog.langchain.dev/what-is-a-cognitive-architecture Wir sind hier
222. #WISSENTEILEN #WISSENTEILEN IMAGE REFERENCE Folie 21: © photoplotnikov - istockphoto.com Folie 23: © Mix und Match Studios - shutterstock.com Folie 23: © Mix und Match Studios - shutterstock.com All other pictures, drawings and icons originate from • pexels.com, • pixabay.com, • unsplash.com, • flaticon.com or are created by my own.