RAGHack-RAG-Copilot-Azure-AI-Studio.pdf

RAG (Retrieval Augmented Generation) is the most common approach used to get LLMs to answer questions grounded in a particular domain's data. Learn how to build a RAG-based custom copilot end-to-end using Azure AI Studio, code-first. We'll walk through "Contoso Chat", a retail copilot scenario with product and customer data. We'll explore prompt engineering using prompty assets, orchestration with promptflow flex-flows and automated provisioning and deployment with azd. You'll learn how to build & test your copilot locally (in VS Code), then deploy & test it in production on Azure.

Relevant Resources
1. #RAGHack Series:
https://aka.ms/raghack
2. RAG In Azure AI Studio Recording: https://reactor.microsoft.com/en-us/reactor/events/23334/
3. Contoso Chat Sample
https://aka.ms/aitour/contoso-chat

Transcript

1. Building RAG-Based Copilots With The Azure AI Platform Nitya Narasimhan, PhD AI Advocacy @Microsoft Leah Bar-On Simmons Azure AI Experiences @ Microsoft
2. Agenda Define: Application Scenario (Contoso Chat)  Provision Azure Infrastructure with AZD  Build RAG-Based Copilot (with your data)  Evaluate Copilot (process & metrics)  Deploy Copilot (hosting & monitoring) Deliver: Responsible & Observable Copilot (Azure AI Studio) https://github.com/microsoft/RAG_Hack/discussions/26
3. Start: Describe the Application Scenario Understand the Application Lifecycle Introduce the core Developer Tools Demo: Contoso Outdoors (UI) with Contoso Chat (AI)
4. Application Scenario: What are we trying to build? Frontend: Contoso Outdoors This is the retailer’s website (chat UI). Customers can ask questions about products or get recommendations Backend: Contoso Chat This is the copilot endpoint (chat AI). It receives chat requests from the UI & returns relevant responses for display.
5. Demo Retail copilot (chat AI) with natural language queries from users Response grounded in product catalog data Response contextual to customer data, history Responsible AI with content safety filters
6. Saf e Rollout/Staging Retrieval Augmente d Generation Hypo thesis Find LLMs Try prompts Operationalizing Building/ augmenting Ideating/ exploring Managing SEND FEEDBACK PREPARE FOR APP DEPLOYMENT ADVANCE PROJECT BUSINESS NEED Deploy LLM App/UI Quo ta and cost management REVERT PROJECT Prompt Engine ering or Fine-tuning Evalua tion Exceptio n Handling C ontent Filtering M onitoring Application Lifecycle: How do we build this in reality?
7. Identify business use case Run flow against sample data Evaluate prompt flow Satisfied? Run flow against larger dataset Evaluate prompt flow Satisfied? Deploy endpoint No No Yes Yes Modify flow (prompts and tools, etc.) Integrate into application Connect to your data and build LLM flows Discover your model Test sample prompt Add monitoring and alerts 1. Ideating/exploring 2. Building/augmenting 3. Operationalizing
8. Developer Experience: What do we care about? Does my app work correctly with diverse user inputs? Evaluation techniques, prompt engineering How can I debug and understand failures in complex workflows? Tracing and debugging, local and production How can I simplify infrastructure & environment setup & maintenance? Infrastructure-as-code templates, Configuration-as-code environments, Unified developer CLI
9. T Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI Developer Tooling: Azure AI Platform & Workflow Tool Task
10. Getting Started: Understand the Design Architecture (RAG) Setup Local Environment (with Codespaces) Provision Azure (with Azure Developer CLI) Demo: Contoso Chat Sample Quickstart Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI
11. Retrieval Augmented Generation: Quick Overview
12. Azure AI Platform: Unified Tooling for Generative AI Best-in-class AI foundation models Azure AI Services Pre-trained, turnkey solutions for intelligent applications Responsible AI Tooling Build and manage apps that are trustworthy by design Azure Machine Learning Full-lifecycle tools for designing and managing AI models Azure AI Studio A comprehensive platform to develop and deploy custom copilots
13. Azure Developer CLI: Code to Cloud with 1 command azd up Azure Developer CLI azd commands Your app The cloud
14. Infrastructure As Code: AI Project Templates for AZD Customer benefits • Quickly deploy applications with the command “azd up” • Common starter AI Scenarios to implement solutions • GitHub Actions for CI/CD deployment and evaluation to test model performance • The CLI allows you to work consistently in a repeatable way across the terminal, your editor/integrated development environment, GitHub Actions pipeline, and more. https://aka.ms/azd-ai-templates
15. Contoso Chat Architecture: An AZD AI Template https://aka.ms/azd-ai-templates
16. Demo Fork Repo – Launch Codespaces – Provision Infra – Validate Setup
17. Build: Prompt engineering with Prompty Orchestration with Prompt flow Demo: Local Development Of Basic Copilot Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Promptflow Promptflow CLI & Evaluators Azure AI Studio UI
18. Prompty Prompty is a language agnostic prompt asset for creating prompts and engineering the responses.
19. Prompty is – a specification + tooling + runtime • Easy to start • Intuitive • Code-first • Language and framework agnostic • Agency with Observability
20. Demo Create Prompty – Configure Prompty – Run Prompty – Iterate Once
21. Contoso Chat Prompty Template – chat.json The “sample” input works for rapid prototyping and testing … but needs orchestration for real data sources
22. End-to-End Development With Prompty @MSBuild
23. Build: Reference multiple data sources (in template) Orchestrate complex flows with Promptys (in code) Demo: Execute chat.prompty with pf CLI – review code Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI
24. Contoso Chat Flow Orchestration: Demo https://aka.ms/azd-ai-templates
25. Demo Execute flow – Explore codebase Template – chat.prompty, chat.json Flow -- chat_request.py, ai_search.py, flex.flow.yaml
26. Contoso Chat local testing – using Prompt flow CLI 1. Test locally with promptflow CLI – specify flow and inputs as parameters 2. Generates a default trace view that can be viewed locally, via browser. 3. On Codespaces, view by clicking the Preview icon for port specified in URL 4. UI for Trace view maintains history of runs, drill-down for per-run details
27. Contoso Chat Trace View – for successful run 1. Open Run View to dive into detailed traces of flow execution .. 2. Integrates traces from openai – get insight into LLM tool execution 3. Get insight into Prompt execution 4. Debug model configuration issues (e.g., LLM Parameters).. And more
28. Contoso Chat Prompty Template – chat.prompty Develop more complex templates with context from multiple data sources, add instructions & safety rules
29. Contoso Chat code – chat_request.py, ai_search.py Complex workflow with multiple models (chat, embedding) & tools (functions) = orchestrated using Azure AI SDKs
30. Evaluate: Understand AI-Assisted Evaluation Flow Understand Quality vs. Safety Metrics Demo: Evaluate copilot with custom quality evaluators Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI
31. AI-Assisted Evaluation – one LLM “grades” the other
34. Evaluation (Preview) – What’s New In Azure AI Studio https://ai.azure.com/build/evaluation
36. Demo (to be updated) Eval dataset - Eval prompty – Eval flow - Eval metrics - Evaluators
37. Deploy: Deploy to Azure AI Studio with AZD Explore Built-in Testing, Integrations Demo: Test the Copilot from AI Studio & from Web App Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI
38. Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate
39. Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate Benefit from integrated dashboards for monitoring and logging to help debug
40. Demo Use `azd deploy` to deploy the application to Azure AI Studio endpoint Use built-in testing with sample inputs to validate Benefit from integrated dashboards for monitoring and logging to help debug Integrate it with Contoso Web UI (Static Web App)
41. Cleanup Use `azd down` to cleanup your resources effectively (with AZD AI templates) Some Azure AI resources have soft delete enabled limiting reuse of names or available quota for models Purge any soft-deleted resources to allow reuse in iterative or collaborative dev environments
42. Let’s Recap
43. What we learned today Provision & Manage Azure Infrastructure Setup & Maintain Local Environment Rapid Prototyping & Prompt Engineering AI-Assisted Flows & Evaluation Metrics Support for Hosting, Safety & Monitoring Azure Developer CLI GitHub Codespaces Prompty & Prompt flow Promptflow CLI & Evaluators Azure AI Studio UI  What is the Application Lifecycle? (LLM Ops)  What is the Azure AI Platform? (Dev Tools)  What is Contoso Chat? (Retail Copilot Sample)  How to build, evaluate, deploy RAG copilot on Azure https://github.com/microsoft/RAG_Hack/discussions/26
44. What you can do to skill up  Explore Other Azure AI Templates https://aka.ms/azd-ai-templates  Fork Contoso Chat and explore/extend it! https://aka.ms/aitour/contoso-chat  Join us on the AI Tour at a city near you! https://aka.ms/aitour https://github.com/microsoft/RAG_Hack/discussions/26