Upgrade to Pro — share decks privately, control downloads, hide ads and more …

SDD 2024 - Large Language Models, data & APIs: ...

SDD 2024 - Large Language Models, data & APIs: integrating generative AI power – with Python & .NET

Human language as a universal interface for software solutions – sounds exciting! In this session, Christian offers an introduction to the integration of generative AI and Large Language Models (LLMs) into your applications. Using Python and .NET APIs, he shows how to exploit the potential of LLMs for various use cases.
At the core of the integration are the attendee’s data, API, and document. We’ll cover architecture patterns like In-Context Learning, Retrieval-Augmented Generation (RAG), Reasoning & Acting (ReAct), and Agents. These techniques are crucial for the development of modern, AI-driven business applications.
For more complex requirements, you’ll see how to employ LangChain for Python and Semantic Kernel for .NET. These frameworks open up enhanced opportunities for text understanding and generation, especially concerning prolonged conversations.
Christian will also provide an overview of Cloud-based solutions, including Azure Open AI Service and Fermyon Serverless AI, focusing on the option to provide and utilize own LLMs based on GPT-4 or Llama2.
Come and join to experience a pragmatic approach to integrating generative AI into business applications!

Christian Weyer

May 14, 2024
Tweet

More Decks by Christian Weyer

Other Decks in Programming

Transcript

  1. Large Language Models, data & APIs: Integrating Generative AI power

    – with Python & .NET Christian Weyer Co-Founder & CTO @christianweyer
  2. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures

    § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Christian Weyer Co-Founder & CTO @ Thinktecture AG 2
  3. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET Our journey today 3 AI all-the- things? LLMs in your Solutions Talk to your Data Exciting Times… Democratizing Generative AI Talk to your Systems
  4. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET AI all-the-things? 6 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP (Natural Language Processing) Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers
  5. § LLMs generate text based on input § LLMs can

    understand text – this changes a lot § Without having to train them on domains or use cases § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Large Language Models (LLMs) 7 Text… – really, just text?
  6. Talk to company knowledge Large Language Models, data & APIs:

    Integrating Generative AI power With Python & .NET DEMO 8
  7. § LLMs are programs § LLMs are highly specialized neural

    networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Large Language Models demystified 9
  8. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET LLMs in your Solutions 10
  9. § LLMs are always part of end-to-end architectures § HTTP/Web/REST

    APIs § Databases § Client apps (Web, desktop, mobile) § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything! Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET End-to-end architectures with LLMs 11
  10. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET Using LLMs: It’s just HTTP APIs Inference, FTW. 12
  11. GPT-4 API access OpenAI Playground Large Language Models, data &

    APIs: Integrating Generative AI power With Python & .NET DEMO 13
  12. § Barebones SDKs § Abstraction over HTTP APIs § E.g.

    Open AI SDK, Mistral SDK § Available for any programming language § Also available from other providers Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Building LLM-based end-to-end applications 14
  13. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET The best tool for any developer to talk to LLMs! 15 🙈
  14. § OSS framework for developing applications powered by LLMs §

    > 1000 contributors § Python and Typescript versions § Chains for sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources § Alternatives like LlamaIndex, Haystack, etc. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET LangChain - building LLM-based applications 16
  15. § Microsoft’s open-source framework to integrate LLMs into applications §

    .NET, Python, and Java versions § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Chain is collection of Plugins § Planners are similar to Agents in LangChain § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Semantic Kernel 17
  16. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET Answering Questions on Data Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 19 Embedding model Embedding model 💡 Indexing / Embedding Question Answering Vector DB
  17. RAG: Chat with web site documents Semantic Kernel, GPT-3.5-Turbo Large

    Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 20
  18. RAG: Learning about company’s policies via Slack LangChain, Weaviate –

    Mixtral-8x7B on Groq Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 21
  19. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET Talk to your Systems 22
  20. § LLM can be instructed to, e.g. § Do sentiment

    analysis § Extract information from text § Extract structured information § Schema description: JSON, TypeScript types, etc. § Implement via tools like Kor, TypeChat, or Open AI Function / Tool Calling Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Extracting meaning in text for structured data 23
  21. Extracting structured data from text: Flight search OpenAI Function Calling,

    LangChain, FastAPI, JS - GPT-4-Turbo Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 24
  22. Extracting structured data from voice: Form filling JSON extraction, OpenAI

    JS SDK, Angular Forms - Mixtral-8x7B on Groq Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 25
  23. § Integrate LLM-external systems to aid LLMs § Tool /

    function calling standard established by OpenAI § LLM outputs JSON containing arguments to call one or many functions § LLM does not call the function § All major libs support tool calling § OpenAI SDKs § LangChain § Semantic Kernel § etc. Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Extending LLM capabilities 26 curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-3.5-turbo", "messages": [ { "role": "user", "content": "What is the weather like in Boston?" } ], "tools": [ { "type": "function", "function": { "name": "get_current_weather", "description": "Get the current weather in a given location", "parameters": { "type": "object", "properties": { "location": { "type": "string", "description": "The city and state, e.g. San Francisco, CA" }, "unit": { "type": "string", "enum": ["celsius", "fahrenheit"] } }, "required": ["location"] } } } ], "tool_choice": "auto" }'
  24. Extending LLM capabilities OpenAI Tool Calling with Semantic Kernel Large

    Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 27
  25. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET Democratizing Generative AI 28
  26. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Open-source Edge IoT Server Desktop Mobile Web 29 Other providers Antrophic Cohere Mistral AI Hugging Face Open-source
  27. § Open-source community drives innovation in Generative AI § Important

    factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Open-source LLMs thrive 30
  28. Local RAG: Mistral-7B open-source LLM llama.cpp, ollama, LangChain, StreamLit Large

    Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 31
  29. Offline in browser: Running open-source LLMs with WebAssembly & WebGPU

    web-llm Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 32
  30. 34 Angular PWA OpenAI Speech-to-Text TT Panorama Gateway OpenAI GPT-4

    OpenAI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data Generate response with availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio TT Panorama Query Panorama API Availability Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET
  31. Ask for experts availability in my company systems Angular, Speech-to-text,

    internal HTTP API, node.js OpenAI SDK + Tool Calling, Text-to-speech – GPT-4-Turbo Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET DEMO 35
  32. § LLMs enable new scenarios & use cases to incorporate

    human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SLMs: specialized, fine-tuned for domains § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Large Language Models, data & APIs: Integrating Generative AI power With Python & .NET Current state 37
  33. Large Language Models, data & APIs: Integrating Generative AI power

    With Python & .NET 2024: The year of SLMs & CPU inference 38