Improve LLM-based Applications with Fallback Mechanisms

Berlin Buzzwords 2024

Talk Description: https://2024.berlinbuzzwords.de/sessions/?id=JMLFV8
YouTube: https://youtu.be/-oYL-FgBkH8?si=Sv0DiuC-tD1YSc6Y

Transcript

1. Improve LLM-based Applications with Fallback Mechanisms Bilge Yücel Berlin Buzzwords 2024
2. About me •🥑 Developer Relations Engineer at deepset •🏗 Open source LLM Framework: Haystack •📍 Istanbul, Turkey Bilge Yücel @bilgeycl Bilge Yücel bilgeyucel
3. Agenda Introduction to Haystack LLMs/RAG Fallback Mechanisms Q&A
4. Large Language Models (LLMs)
5. Retrieval Augmented Generation
6. Retrieval Augmented Generation
7. Retrieval Augmented Generation
9. • Haystack is an open source Python framework for building production-ready LLM applications
10. • Haystack is an open source Python framework for building production-ready LLM applications • Prototyping, evaluation, deployment, monitoring…
11. • Haystack is an open source Python framework for building production-ready LLM applications • Prototyping, evaluation, deployment, monitoring… • Building blocks: Components & Pipelines
12. Haystack
13. Haystack
14. RAG PIPELINE
15. RAG PIPELINE
16. RAG PIPELINE
17. RAG PIPELINE
18. RAG PIPELINE
19. RAG PIPELINE
21. Fallback Mechanisms
22. Use Cases Extracting Structured Data Early Routing with Fallback RAG with Web Fallback
23. RAG PIPELINE When is Taylor Swift’s next concert?
24. RAG PIPELINE When is Taylor Swift’s next concert? The documents do not provide any information about Taylor Swift's upcoming concert.
25. Mechanism: Routing
26. Routing - Prompt Template
27. Routing - Conditional Router
28. Fallback to Web
29. Fallback to Web When is Taylor Swift’s next concert?
30. Fallback to Web When is Taylor Swift’s next concert? Taylor Swift's next concert is on June 13, 2024, in Liverpool.
31. Fallback to Other Sources
32. Early Routing
33. Early Routing
34. Early Routing
35. Early Routing (Tools)
36. Early Routing (Tools)
37. Early Routing (Tools)
38. Early Routing (Tools)
39. Early Routing (Tools)
40. Early Routing (Tools)
41. Early Routing - Conditional Router
42. Early Routing (Tools)
43. Early Routing (Tools) "How should I use the OpenAIChatGenerator?"
44. Early Routing (Tools) '{"response": "search_haystack"}' "How should I use the OpenAIChatGenerator?"
45. Early Routing (Tools) "When is Berlin Buzzwords 2024?"
46. Early Routing (Tools) '{"response": "search_web"}' "When is Berlin Buzzwords 2024?"
47. Early Routing (Tools) "What's the first name of Einstein?"
48. Early Routing (Tools) '{"response": "none"}' "What's the first name of Einstein?"
49. Early Routing (Tools)
50. Extracting Structured Data
51. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
52. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
53. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
54. Extracting Structured Data Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
55. Mechanism: Looping
56. OutputValidator
57. OutputValidator
58. Prompt Template
59. Pipeline code
60. Pipeline code
61. Pipeline code
62. Pipeline code
63. Pipeline code
64. Looping Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
65. Looping Berlin is the capital of Germany. It has a population of 3,850,809. Paris, France's capital, has 2.161 million residents. Lisbon is the capital and the largest city of Portugal with the population of 504,718.
66. Recap • Fallbacks are handy to make a search in alternative sources • Fallback branches can be action triggers • Early routing is also possible if you have defined options • Loops are great for validation and auto-correction
67. Sources 🌐 Fallback to Web 🔄 Generating Structured Output 🌳 Early Routing
68. Thank you @bilgeycl Bilge Yücel bilgeyucel haystack.deepset.ai Haystack is open source! ⭐ 📖 Documentation 📚 Tutorials 󰳏 Cookbooks 📘 Articles 👾 Join the Community