AI Chat App Hack: Building a RAG Chat App

Slides from first session of https://github.com/microsoft/AI-Chat-App-Hack

Transcript

3. Building a RAG Chat App with Azure AI Search, OpenAI, and Python
4. Goal Build an AI Chat App using the RAG approach to combine an LLM with your own data.
5. LLMs
6. LLM: Large Language Model An LLM is a model that is so large that it achieves general-purpose language understanding and generation. Review: This movie sucks. Sentiment: negative Review: I love this movie: Sentiment: Input LLM positive Output
7. LLMs in use today Model # of Parameters Creator Uses GPT 3.5 175 B OpenAI ChatGPT, Copilots, APIs GPT 4 Undisclosed OpenAI PaLM 540 B Google Bard Claude 2 130 B Anthropic APIs LlaMA 70 B Meta OSS Mistral-7B 7 B Mistral AI OSS
8. GPT: Generative Pre-trained Transformer Learn more: •Andrej Karpathy: State of GPT •Andrej Karpathy: Let's build GPT: from scratch, in code GPT models are LLMs based on Transformer architecture from "Attention is all you need" paper
9. Using OpenAI GPT models: Azure Studio System message + User question = Chat Completion response
10. Using OpenAI GPT models: Python response = openai.ChatCompletion.create( stream=True, messages = [ { "role": "system", "content": "You are a helpful assistant with very flowery language" }, { "role": "user", "content": "What food would magical kitties eat?” } ]) for event in response: print(event.choices[0].delta.content)
11. The limitations of LLMs Outdated public knowledge No internal knowledge
12. Incorporating domain knowledge Prompt engineering Fine tuning Retrieval Augmented Generation In-context learning Learn new skills (permanently) Learn new facts (temporarily)
13. Retrieval Augmented Generation
14. RAG: Retrieval Augmented Generation Document Search PerksPlus.pdf#page=2: Some of the lessons covered under PerksPlus include: · Skiing and snowboarding lessons · Scuba diving lessons · Surfing lessons · Horseback riding lessons These lessons provide employees with the opportunity to try new things, challenge themselves, and improve their physical skills.…. Large Language Model Yes, your company perks cover underwater activities such as scuba diving lessons 1 User Question Do my company perks cover underwater activities?
15. The benefit of RAG Up-to-date public knowledge Access to internal knowledge
16. RAG components Component Examples Retriever: A knowledge base that can efficiently retrieve sources that match a user query Azure AI Search, Azure CosmosDB, PostgreSQL, Qdrant, Pinecone LLM: A model that can answer questions based on the query based on the provided sources, and can include citations GPT 3.5, GPT 4 Glue: A way to chain the retriever to the LLM (optional) Langchain, Llamaindex, Semantic Kernel Features Chat history, Feedback buttons, Text-to-speech, User login, File upload, Access control, etc.
17. Many ways to build a RAG chat app No Code Low Code High Code Copilot studio Azure Studio On Your Data github.com/ azure-search-openai-demo
18. Copilot Studio – On Your Data Retriever: Uploaded files LLM: GPT 3.5 https://copilotstudio.preview.microsoft.com/
19. Azure Studio – On Your Data Retriever: Azure AI Search Azure Blob Storage Azure CosmosDB for MongoDB vCore URL/Web address Uploaded files LLM: GPT 3.5/4 Features: User authentication Chat history persistence https://learn.microsoft.com/azure/ai-services/openai/concepts/use-your-data
20. Open source RAG chat app solution Retriever: Azure AI Search LLM: GPT 3.5/4 Features: Multi-turn chats User authentication with ACLs Chat with image documents https://github.com/Azure-Samples/azure-search-openai-demo/ aka.ms/ragchat
21. Deep dive: RAG chat app solution
22. Prerequisites • Azure account and subscription • A free account can be used, but will have limitations. • Access to Azure OpenAI or an openai.com account • Request access to Azure OpenAI today! https://aka.ms/oaiapply https://github.com/Azure-Samples/azure-search-openai-demo/#azure-account-requirements
23. Opening the project: 3 options • GitHub Codespaces → • VS Code with Dev Containers extension • Your Local Environment • Python 3.9+ • Node 14+ • Azure Developer CLI https://github.com/Azure-Samples/azure-search-openai-demo/?tab=readme-ov-file#project-setup
24. Deploying with the Azure Developer CLI azd auth login --use-device-code azd env new azd up Login to your Azure account: Create a new azd environment: (to track deployment parameters) Provision resources and deploy app: azd up is a combination of azd provision and azd deploy
25. Application architecture on Azure Azure Storage Document Intelligence Integrated vectorization or Local script Azure OpenAI Azure AI Search Uploads PDF pages Computes embeddings Stores in index Extracts data from PDFs DATA INGESTION Splits data into chunks Python Azure OpenAI Azure App Service or Local server Azure Storage CHAT APP Azure AI Search
26. Code walkthrough Typescript frontend (React, FluentUI) Python backend (Quart, Uvicorn) chat.tsx makeApiRequest() api.ts chatApi() app.py chat() chatreadretrieveread.py run() get_search_query() compute_text_embedding() search() get_messages_from_history() chat.completions.create()
27. Search approach Vector Keywords Fusion (RRF) Reranking Learn more at this week’s session: Azure AI Search Best Practices For optimal retrieval, search() uses hybrid retrieval (text + vectors) plus the semantic ranker option. https://aka.ms/ragrelevance
28. Next steps • Register for the hackathon → • Introduce yourself in our discussion forum • Deploy the repo with the sample data • See steps on low cost deployment → • Post in forum if you have any questions or issues deploying. • Join tomorrow’s session: Customizing your RAG Chat App! aka.ms/hacktogether/chatapp aka.ms/ragchat/free