Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Microsoft Build 2023 Updates – Copilot Stack an...

Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)

第78回 Machine Learning 15minutes! Broadcast (2023/06/24)
https://machine-learning15minutes.connpass.com/event/285147/

S/N Ratio > Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI Service (Machine Learning 15minutes! Broadcast #78)
https://satonaoki.wordpress.com/2023/06/24/build-2023-updates-copilot-stack-azure-openai/

SATO Naoki (Neo)

June 24, 2023
Tweet

More Decks by SATO Naoki (Neo)

Other Decks in Technology

Transcript

  1. Microsoft Build 2023 Updates – Copilot Stack and Azure OpenAI

    Service SATO Naoki (Neo) Senior Software Engineer, Microsoft
  2. +

  3. World’s most powerful supercomputers World’s most capable foundation models, from

    hosted to open source World’s best AI dev infrastructure
  4. Augment AI systems to access APIs Retrieve useful information Perform

    new computations Safely act on the user’s behalf Plugins
  5. Copilot stack AI infrastructure Foundation models AI safety BYO models

    Hosted fine-tuned foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX
  6. Foundation models AI safety BYO models Hosted fine-tuned foundation models

    Hosted foundation models Plugin extensibility Copilot frontend UX Orchestration frameworks Copilot stack Plugin execution Grounding Orchestration Metaprompt Prompt & response filtering
  7. Foundation models AI safety Plugin extensibility Plugin execution Grounding Orchestration

    Copilot frontend UX Metaprompt Prompt & response filtering Prompt and Metaprompt Copilot stack
  8. Metaprompt Foundation models AI safety BYO models Hosted fine-tuned foundation

    models Hosted foundation models Plugin extensibility Plugin execution Orchestration UX AI infrastructure Vector databases • Web APIs • Plugins Prompt & response filtering Copilot stack Retrieval Augmented Generation (RAG) Grounding
  9. Foundation models AI safety Grounding Metaprompt Prompt & response filtering

    Orchestration Copilot frontend UX Plugin execution Plugin extensibility Copilot stack Plugins
  10. AI safety Plugin execution Grounding Metaprompt Prompt & response filtering

    AI infrastructure Foundation models BYO models Hosted fine-tuned foundation models Hosted foundation models Copilot stack Foundation models and fine-tuning
  11. AI infrastructure Foundation models AI safety BYO models Hosted fine-tuned

    foundation models Hosted foundation models Plugin extensibility Plugin execution Grounding Metaprompt Prompt & response filtering Orchestration Copilot frontend UX Copilot stack
  12. Azure AI Applied AI Services Bot Service Cognitive Search Form

    Recognizer Video Indexer Metrics Advisor Immersive Reader Cognitive Services Vision Speech Language Decision Azure OpenAI Service Azure Machine Learning Prepare & Preprocess Build, Train & Consume Deploy & Scale Manage & Monitor AI Infrastructure
  13. Announcing Azure AI Studio Build and train your own models

    Ground Azure OpenAI Service and OSS models using your data Built-in vector indexing Retrieval augmented generation made easy Create prompt workflows AI safety built-in
  14. Generative AI Applications Azure Machine Learning Native OSS Model Catalog

    Prompt Eng/Eval Prompt flow Responsible AI Azure Content Safety High Scale Gen AI App Deployment Gen AI Model Monitoring
  15. Radically changing the art of possible with Azure OpenAI Service

    Large pretrained foundation AI models custom-tunable with your parameters and your data Summarization Reasoning over data Writing tools Code generation ChatGPT The Era of Copilots GPT-3 (GA) DALL•E 2 (preview) ChatGPT (GA) GPT-4 (GA) Foundation of enterprise security, privacy and compliance
  16. Update Bookmark ‘what's new’ in Azure OpenAI Service Sign up

    for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June now Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June now Provisioned Throughput Limited Availability early June
  17. Anatomy of a RAG app App UX Orchestrator Retriever over

    Knowledge Base Query → Knowledge Prompt + Knowledge → Response Large Language Model Build your own experience UX, orchestration, calls to retriever and LLM e.g., Copilots, in-app chat Extend other app experiences Plugins for retrieval, symbolic math, app integration, etc. e.g., plugins for OpenAI ChatGPT
  18. Retrievers: Externalizing Knowledge “Find the most relevant snippets in a

    large data collection, using unstructured input as query” == search engine App UX Orchestrator Azure OpenAI Azure Cognitive Search Data Sources (files, databases, etc.) Query → Knowledge Prompt + Knowledge → Response Azure Cognitive Search  Azure’s complete retrieval solution  Data ingestion, enterprise-grade security, partitioning and replication for scaling, support for 50+ written languages, and more
  19. Retrieving Using Semantic Similarity Vector representations (or embeddings)  Learned

    such that “close” vectors represent items with similar meaning  May encode words, sentences, images, audio, etc.  Some map multiple media types into the same space  Azure OpenAI embeddings API, OSS embeddings (e.g., SBERT, CLIP)
  20. Vector-based Retrieval Encoding (vectorizing)  Pre-process and encode content during

    ingestion  Encode queries during search/retrieval Vector indexing  Store and index lots of n-dimensional vectors  Quickly retrieve K closest to a “query” vector  Exhaustive search impractical in most cases  Approximate nearest neighbor (ANN) search Embedding [0.023883354, 0.021508986, 0.044205155, 0.019588541, 0.031198505, …]
  21. Vector Search in Azure Cognitive Search New vector type for

    index fields  Users indicate vector size, distance function, algorithm and algo-specific parameters Pure Vector Search & Hybrid Search  Filters, faceting, etc. all works with vectors  Integrates with existing search indexes  Existing data ingestion and augmentation machinery entirely applicable Combines well with L2 re-ranker powered by Bing’s models  Enables improved ranking for hybrid search scenarios  L1: keywords + vector retrieval  L2: Bing’s ranker refreshed with GPT-enhanced work Enterprise-grade  Scalability (partitioning, replication)  Security: network isolation, managed identities, RBAC, etc.
  22. Revolutionizing Indexing and Retrieval for LLM-powered Apps Power your retrieval-augmented

    generation applications Images Audio Video Graphs Documents • Use vector or hybrid search • Use Azure OpenAI embeddings or bring your own • Deeply integrate with Azure • Scale with replication and partitioning • Build generative AI apps and retrieval plugins Sign up today https://aka.ms/VectorSearchSignUp
  23. Chat Completions API Versatile interface use for all scenarios—not just

    chat Model adheres to instructions in “system” message It sets the behavioral guidelines for the model, including responsible AI steering Examples go into “User”, “Assistant” fields import openai openai.api_type = "azure” response = openai.ChatCompletion.create( engine ="gpt-4", messages=[ {"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Who won the world series in 2020?"}, {"role": "assistant", "content": "The LA Dodgers won the World Series in 2020."}, {"role": "user", "content": "Where was it played?"} ] ) …
  24. Using your data Challenge I want to reason over my

    very long documents, but the token limits are not sufficient Conventional solution Build a vector database Retrieve relevant data and construct prompt at run-time
  25. Introducing: Azure OpenAI Service on your data (Preview) Data Sources

    (search, files, databases, storage etc.) Additional 3P Data Sources (future capabilities) Azure OpenAI Service on your data API & SDK App or Copilot agent
  26. Expanding potential Challenges we wanted to address Accurate translation for

    wider range of languages—improve support for Asian and African languages Integrate vector databases and cloud data stores Use up-to-date information from the web
  27. Introducing: Azure OpenAI Service Plugins (coming soon) Build powerful AI

    Copilots with secure access to Microsoft services Retrieve data with Azure Cognitive Search Translate >100 languages with Azure Translator Ground with recent info with Bing Search Extract structured data from Azure SQL Azure OpenAI Plugins • Securely access your data in various data stores, vector databases and the web • Data path access controlled via Azure AD and Managed Identities • Admin roles to choose what plugins to enable
  28. Recap Bookmark ‘what's new’ in Azure OpenAI Service Sign up

    for Azure OpenAI Service updates Azure OpenAI Service GPT-3 (GA) DALL·E 2 (preview) ChatGPT (GA) GPT-4 (GA) Apply your own data Available in Preview early June Plugins for Azure OpenAI Service Coming soon Configurable Content Filters Available in Preview early June Provisioned Throughput Limited Availability early June
  29. Resources  Microsoft Build (2023/05/23-25)  https://build.microsoft.com/  DEEP LEARNING

    LAB - [Recap] Microsoft Build 2023 最新アップデー トAnalytics&AI (2023/06/08)  https://dllab.connpass.com/event/284571/  Microsoft Build 2023 Azure AI&ML 最新アップデート - Speaker Deck  https://speakerdeck.com/shohei1029/microsoft-build-2023-azure-ai-and-ml-zui-xin-atupudeto  Microsoft Build Japan (2023/06/27‐28)  https://info.microsoft.com/JA-ADAI-CATALOG-FY23-06Jun-28-Microsoft-Build-Japan-Day2- SREVM14500_Catalog-Display-Page.html