Share
Webinar: Large Language Models, Daten & APIs: Integration von Generative AI Power – mit Python & .NET
Jenseits des ChatGPT-Hypes taucht Christian in die Welt der Large Language Models (LLMs), Daten und APIs ein und konzentriert sich darauf, wie Sie AI-Funktionalität sinnvoll in Ihre Anwendungenintegrieren können. Wir werden pragmatische Szenarien und Use Cases untersuchen, die das Potenzial von LLMs (wie GPT oder Llama) demonstrieren - und erörtern, wie AI-Techniken in bestehende Architekturen einbezogen werden können. Die Teilnehmer erhalten erste Einblicke sowohl in LangChain als Python-Toolkit als auch Semantic Kernel als .NET-basiertes SDK.
Zudem werden wir darauf eingehen, nicht nur Closed-Source-Systeme (wie OpenAI) zu nutzen, sondern auch Open-Source-Optionen in Betracht zu ziehen, um unterschiedlichen Anforderungen gerecht werden zu können.
Christian Weyer PRO
Transcript
35 slides
-
Large Language Models, Daten & APIs: Integration von Generative AI
Power – mit Python & .NET Christian Weyer @christianweyer CTO, Technology Catalyst -
§ What & why? § LLM integration aspects § Selected
use cases § Agents & LLMs § LLM landscape § Current state & Outlook Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Talking points 2 -
§ Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures
§ Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Christian Weyer Co-Founder & CTO @ Thinktecture AG 3 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Intro 4 -
Speech OpenAI Whisper tortoise-tts … Large Language Models, Daten &
APIs Integration von Generative AI Power - mit Python & .NET Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 5 -
Speech OpenAI Whisper tortoise-tts … Large Language Models, Daten &
APIs Integration von Generative AI Power - mit Python & .NET Generative AI all-the-things Images Midjourney DALL·E … Audio/Music Musico Soundraw … Text OpenAI GPT Anthropic Claude Codex Llama … 6 P -
§ LLMs generate text based on input § LLMs can
understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 * LLMs are not “perfect” – errors may occur, caveats like non-determinism & hallucination – these are topics to be dealt with Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Text… – really, just text? 7 Yeah: Large Language Models (LLMs) -
“Hm… what … how… erm… who could know this…? 🤔”
Learning about my company’s policies in Slack Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET DEMO 8 Demos are using OpenAI GPT-3.5-turbo or GPT-4 -
§ LLMs are programs § LLMs are highly specialized neural
networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Large Language Models demystified 9 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Integration 10 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Using LLMs: It’s just APIs ! Inference, FTW. 11 -
GPT-4 API access via OpenAI Playground Large Language Models, Daten
& APIs Integration von Generative AI Power - mit Python & .NET DEMO 12 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET The best tool for .NET developers to talk to LLMs! 13 🙈 -
§ Most popular open-source framework for developing applications powered by
LLMs § Several releases per week § > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of actions in code § Abstractions for § Loading text from a wide range of sources § Vector stores § Prompts & LLMs (local and remote) § Tools and Agents Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET LangChain 14 -
§ Microsoft’s open-source framework to integrate LLMs into applications §
.NET, Python, and Java versions § Plugins encapsulate AI capabilities into a single unit of functionality § Semantic functions for prompting § Native functions to run local code § Chain is collection of Plugins § Planners are similar to Agents in LangChain § Not as broad feature set as LangChain § E.g., no concept/abstraction for loading data Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Semantic Kernel 15 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Use cases 16 -
Text generation § LLMs are good in generating text §
Regular text § Code § SQL (beware!) § JSON § etc. Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Typical LLM use cases 17 -
Generating text & images (multimodal) (Semantic Kernel) Large Language Models,
Daten & APIs Integration von Generative AI Power - mit Python & .NET DEMO 18 -
Understanding text § LLM can be instructed to e.g. §
do sentiment analysis § extract information from text § Tools (like Kor, Open AI Functions – or TypeChat) for extracting information in a structured way reliably § JSON, Python classes, TypeScript types etc. Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Typical LLM use cases 19 -
Extracting structured data (LangChain + Kor) Large Language Models, Daten
& APIs Integration von Generative AI Power - mit Python & .NET DEMO 20 -
Answering Questions / Chat with documents § Mixed usage of
text embedding and a large language model § Retrieval-augmented generation (RAG) § Index one or more documents § Use text embedding to index documents § Store embeddings in a vector database § User asks question § Create embedding for the question § Search vector database to find relevant text § Use LLM with relevant input text & question to create an answer Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Typical LLM use cases 21 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Answering Questions / Chat with documents - RAG Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 22 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA -
Chat with web site documents (Semantic Kernel) Large Language Models,
Daten & APIs Integration von Generative AI Power - mit Python & .NET DEMO 23 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Agents 24 -
§ Involve an LLM making decisions (via ReAct – Reasoning
& Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Agents: Multi-steps workflows with LLM interaction 25 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET ReAct – with function calling 26 LLM My code Query Some API Some database Prompt Function definitions Final answer Answer ❓ ❓ ❗ 💡 -
Stock market agent with OpenAI function calling (Python OpenAI SDK)
Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET DEMO 27 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Landscape 28 -
§ Open AI § Azure Open AI Service § Google
Model Garden on Vertex AI § Amazon Sagemaker § Hugging Face § Fermyon Serverless AI § Self-hosting (hard to scale) § Device / browser Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET LLM providers 29 -
Running an open-source LLM offline in the browser (with Wasm)
(web-llm) Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET DEMO 30 -
Large Language Models, Daten & APIs Integration von Generative AI
Power - mit Python & .NET Wrap-Up 31 -
§ Fast moving and changing field § Every week something
“big” happens in LLM space § Frameworks are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § Bias of the models § Which data was the model trained on? § Finding the right prompt can be tedious § Prompting differs between LLMs § Prompt injection as a threat § Will improve over time § Context size is currently limited § Will likely diminish over time § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Current state 32 -
Huge potential for LLM-driven human-machine workflows via UI (universal interface)
agents Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Outlook 33 P -
§ Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ §
LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf Large Language Models, Daten & APIs Integration von Generative AI Power - mit Python & .NET Links 35