Large Language Models, Data & APIs: Integrating Generative AI Power - with Python & .NET

Let's look beyond the ChatGPT hype! In this session, Christian delves into the world of Large Language Models (LLMs), data, and APIs, focusing on how you can meaningfully integrate AI functionality into your applications. We will explore pragmatic scenarios and use cases that demonstrate the potential of LLMs (such as GPT or Llama) and discuss how AI techniques can be incorporated into existing architectures. Participants will gain initial insights into LangChain as a Python toolkit and Semantic Kernel as a .NET-based SDK. Additionally, we will address the consideration of not only using closed-source systems (like OpenAI) but also taking open-source options into account to meet diverse requirements. Come in and find out!

Transcript

1. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Christian Weyer @christianweyer CTO, Technology Catalyst
2. § What & why? § LLM integration aspects § Selected use cases § Agents & LLMs § LLM landscape § Current state & Outlook Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Talking points 2
3. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies christian.weyer@thinktecture.com @christianweyer https://www.thinktecture.com Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Christian Weyer Co-Founder & CTO @ Thinktecture AG 3
4. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Intro 4
5. Speech OpenAI Whisper tortoise-tts … Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 5
6. Speech OpenAI Whisper tortoise-tts … Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 6
7. § LLMs generate text based on input § LLMs can understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 * LLMs are not “perfect” – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Text… – really, just text? 7 Yeah: Large Language Models (LLMs)
8. “Hm… what … how… erm… who could know this…? 🤔” Learning about my company’s policies in Slack Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 8 Demos are using OpenAI GPT-3.5-turbo or GPT-4
9. § LLMs are programs § LLMs are highly specialized neural networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Large Language Models demystified 9
10. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Integration 10
11. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Using LLMs: It’s just APIs ! Inference, FTW. 11
12. GPT-4 API access via OpenAI Playground Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 12
13. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET The best tool for .NET developers to talk to LLMs! 13 🙈
14. § Most popular open-source framework for developing applications powered by LLMs § Several releases per week § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of actions in code § Abstractions for § Prompts & LLMs (local and remote) § Vector stores § Tools and Agents § Loading text from a wide range of sources Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET LangChain 14
15. § Microsoft’s open-source framework to integrate LLMs into applications § .NET, Python, and Java versions § Plugins encapsulate AI capabilities § Semantic functions for prompting § Native functions to run local code § Chain is collection of Plugins § Planners are similar to Agents in LangChain § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Semantic Kernel 15
16. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Use cases 16
17. Text generation § LLMs are good in generating text § Regular text § Code § SQL (beware!) § JSON § etc. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Typical LLM use cases 17
18. Generating text & images (Semantic Kernel) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 18
19. Understanding text § LLM can be instructed to e.g. § do sentiment analysis § extract information from text § Tools (like Kor, Open AI Functions – or TypeChat) for extracting information in a structured way reliably § JSON, Python classes, TypeScript types etc. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Typical LLM use cases 19
20. Extracting structured data (LangChain + Kor) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 20
21. Answering Questions / Chat with documents § Mixed usage of text embedding and a large language model § Retrieval-augmented generation (RAG) § Index one or more documents § Use text embedding to index documents § Store embeddings in a vector database § User asks question § Create embedding for the question § Search vector database to find relevant text § Use LLM with relevant input text & question to create an answer Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Typical LLM use cases 21
22. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Answering Questions / Chat with documents - RAG Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 22 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA
23. Chat with web site documents (Semantic Kernel) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 23
24. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Agents 24
25. § Involve an LLM making decisions (via ReAct – Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Agents: Multi-steps workflows with LLM interaction 25
26. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET ReAct – with function calling 26 LLM My code Query Some API Some database Prompt Function definitions Final answer Answer ❓ ❓ ❗ 💡
27. Stock market agent with OpenAI function calling (Python OpenAI SDK) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 27
28. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Landscape 28
29. § Open AI § Azure Open AI Service § Google Model Garden on Vertex AI § Amazon Sagemaker § Hugging Face § Fermyon Serverless AI § Self-hosting (hard to scale) § Device / browser Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET LLM providers 29
30. Running an open-source LLM offline in the browser (with Wasm) (web-llm) Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET DEMO 30
31. Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Wrap-Up 31
32. § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § Bias of the models § Which data was the model trained on? § Finding the right prompt can be tedious § Prompting differs between LLMs § Prompt injection as a threat § Will improve over time § Context size is currently limited § Will likely diminish over time § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Current state 32
33. Huge potential for LLM-driven human-machine workflows via UI (universal interface) agents Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Outlook 33
34. Thank you! Christian Weyer https://thinktecture.com/christian-weyer 34
35. § Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ § LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf Large Language Models, Data & APIs Integrating Generative AI Power - with Python & .NET Links 35