Incorporate Fallback Mechanisms For Your Amazon Bedrock RAG Pipelines

My presentation at AWS Community Day Turkey 2024

Github repo with colab notebooks: https://github.com/bilgeyucel/presentations/tree/main/aws-community-day-2024

Large Language Model (LLM)-based systems have demonstrated remarkable advancements in various natural language processing (NLP) tasks, particularly through the Retrieval Augmented Generation (RAG) approach. This approach addresses some of the pitfalls associated with LLMs, such as hallucinations or issues related to the recentness of its training data. However, RAG systems may encounter other challenges in real-world scenarios, including handling out-of-domain queries (e.g., requesting medical advice from a finance app), struggling to generate meaningful answers from poorly retrieved data, or failing to provide any answer at all. To address these situations effectively, it is necessary to implement a fallback mechanism capable of gracefully handling such scenarios. This fallback mechanism can incorporate alternative strategies, such as conducting a web search with the same query to retrieve more up-to-date information or utilizing alternative information sources (such as Slack, Notion, Google Drive, etc.) to gather more relevant data and generate a satisfactory or comprehensive response. However, the question arises: how can we determine if the response is inadequate? During this session, we will start from scratch and lay the groundwork for generative AI applications. Next, we will explore various fallback mechanism techniques and ensure that our system can assess the adequacy of a response and improve it if necessary without human intervention. On the practical side, we will use the open-source LLM framework Haystack and Amazon Bedrock foundation models to implement end-to-end RAG systems. By the end of this talk, you will have learned to select the appropriate fallback method for your use case, enabling you to develop more dependable and versatile LLM-based systems and implement them effectively using Amazon Bedrock and Haystack.

Transcript

1. Incorporate Fallback Mechanisms For Your Amazon Bedrock RAG Pipelines Bilge Yücel AWS Community Day Türkiye 2024
2. About me •🥑 Developer Relations Engineer at deepset 󰎲 •🏗 Open source LLM Framework: Haystack •🎓 Sabancı University B.Sc. •🧠 KU Leuven M.Sc. 󰎐 •📍 Istanbul, Turkey Bilge Yücel @bilgeycl Bilge Yücel
3. Agenda Haystack & Amazon Bedrock Retrieval Augmented Generation Fallback Mechanisms Q&A
4. Large Language Models (LLMs)
5. Retrieval Augmented Generation
6. Retrieval Augmented Generation
7. Retrieval Augmented Generation
9. • Haystack is an open source Python framework for building production-ready LLM applications
10. • Haystack is an open source Python framework for building production-ready LLM applications • Prototyping, evaluation, deployment, monitoring…
11. • Haystack is an open source Python framework for building production-ready LLM applications • Prototyping, evaluation, deployment, monitoring… • Building blocks: Components & Pipelines
12. Haystack
13. Haystack
14. • Foundation models from Anthropic, Cohere, Meta, Mistral AI, Amazon and more through single API Amazon Bedrock
15. • Foundation models from Anthropic, Cohere, Meta, Mistral AI, Amazon and more through single API • Serverless, no infrastructure management Amazon Bedrock
16. • Foundation models from Anthropic, Cohere, Meta, Mistral AI, Amazon and more through single API • Serverless, no infrastructure management • Data is not shared with model providers Amazon Bedrock
17. • Foundation models from Anthropic, Cohere, Meta, Mistral AI, Amazon and more through single API • Serverless, no infrastructure management • Data is not shared with model providers • Haystack supports it! Amazon Bedrock
18. Models
19. RAG PIPELINE
20. RAG PIPELINE
21. RAG PIPELINE
22. RAG PIPELINE
23. RAG PIPELINE
24. RAG PIPELINE
26. Fallback Mechanisms Looping Routing/Branching
27. RAG PIPELINE When is Taylor Swift’s next concert?
28. RAG PIPELINE When is Taylor Swift’s next concert? The documents do not provide any information about Taylor Swift's upcoming concert.
29. Routing
30. Routing - Prompt Template
31. Routing - Conditional Router
32. Fallback to Web
33. Fallback to Web When is Taylor Swift’s next concert?
34. Fallback to Web When is Taylor Swift’s next concert? Taylor Swift's next concert is on May 17, 2024, at Friends Arena in Solna, Sweden.
35. Fallback to Other Sources
36. Extracting Structured Data
37. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
38. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
39. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
40. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
41. Looping
42. OutputValidator
43. Prompt Template
44. Looping Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
45. Looping Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
46. Sources 🔄 Generating Structured Output 🔗 🌐 Fallback to Web 🔗
47. Teşekkürler @bilgeycl Bilge Yücel bilgeyucel haystack.deepset.ai Haystack is open source! ⭐