Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generative AI: Integration von Large Language M...

Generative AI: Integration von Large Language Models in eigene Software - APIs, Daten & Code

Jajaja, der ChatGPT-Wahnsinn. Einfach Text in ein Chat-Fenster tippen und staunen ... Jo, nice. Aber was ist das wirklich Interessante daran für uns Softwarearchitekten und -entwickler? Christian zeichnet in dieser Night Session jenseits des aktuellen Hypes ein Bild, in dem Large Language Models (LLMs) und die menschliche Sprache ein wichtiger und zentraler Bestandteil zukünftiger Softwarelösungen sein kann. Die Mächtigkeit semantischer Schnittstellen via LLMs, gepaart mit klassischen Daten-Architekturen, APIs und Workflows, eröffnet interessante Use Cases für die Mensch-Maschine- respektive Mensch-Code-Interaktion. Die Session bringt die neuen Ideen von generativer AI & Large Language Models in Einklang mit existierenden Architekturansätzen, damit wir als Developer das mögliche Potenzial für künftige Lösungen und Produkte besser einschätzen können.

Christian Weyer

December 04, 2023
Tweet

More Decks by Christian Weyer

Other Decks in Programming

Transcript

  1. Generative AI: Integration von Large Language Models in eigene Software

    - APIs, Daten & Code Christian Weyer @christianweyer CTO, Technology Catalyst
  2. § Technology catalyst § AI-powered solutions § Pragmatic end-to-end architectures

    § Microsoft Regional Director § Microsoft MVP for Developer Technologies & Azure ASPInsider, AzureInsider § Google GDE for Web Technologies [email protected] @christianweyer https://www.thinktecture.com Integration von Large Language Models in eigene Software APIs, Daten & Code Christian Weyer Co-Founder & CTO @ Thinktecture AG 2
  3. Integration von Large Language Models in eigene Software APIs, Daten

    & Code Our journey 3 (Gen) AI all- the-things? Integrating LLMs Selected Scenarios Democratizing Generative AI Exciting Times… Agents
  4. Integration von Large Language Models in eigene Software APIs, Daten

    & Code AI all-the-things? 5 Data Science Artificial Intelligence Machine Learning Unsupervised, supervised, reinforcement learning Deep Learning ANN, CNN, RNN etc. NLP Generative AI GAN, VAE, Transformers etc. Image / Video Generation GAN, VAE Large Language Models Transformers
  5. § LLMs generate text based on input § LLMs can

    understand text – this changes a lot § Prompts are the universal interface (“UI”) → unstructured text with semantics § Human language evolves as a first-class citizen in software architecture 🤯 Integration von Large Language Models in eigene Software APIs, Daten & Code Large Language Models (LLMs) 6 Text… – really, just text?
  6. § LLMs are programs § LLMs are highly specialized neural

    networks § LLMs use(d) lots of data § LLMs need a lot of resources to be operated § LLMs have an API to be used through Integration von Large Language Models in eigene Software APIs, Daten & Code Large Language Models demystified 7
  7. § LLMs are always part of end-to-end architectures § HTTP/Web/REST

    APIs § Databases § Client apps (Web, desktop, mobile) § etc. § An LLM is ‘just’ an additional asset in your architecture § It is not the Holy Grail for everything! Integration von Large Language Models in eigene Software APIs, Daten & Code End-to-end architectures with LLMs 9
  8. Integration von Large Language Models in eigene Software APIs, Daten

    & Code Using LLMs: It’s just APIs ! Inference, FTW. 10
  9. GPT-4 API access via OpenAI Playground Integration von Large Language

    Models in eigene Software APIs, Daten & Code DEMO 11
  10. Integration von Large Language Models in eigene Software APIs, Daten

    & Code The best tool for developers to talk to LLMs! 12
  11. § OSS framework for developing applications powered by LLMs §

    > 1000 contributors § Python and Typescript versions § Chains as a central concept § Sequences of LLM-related actions in code § Abstractions for § Prompts & LLMs (local and remote) § Memory § Vector stores § Tools § Loading text from a wide range of sources Integration von Large Language Models in eigene Software APIs, Daten & Code E.g.: LangChain - building LLM-based applications 13
  12. Text generation § LLMs are good in generating text §

    Regular text § Code § SQL (beware!) § JSON § etc. Integration von Large Language Models in eigene Software APIs, Daten & Code Typical LLM scenarios: 15
  13. Extracting meaning in text § LLM can be instructed to,

    e.g. § Do sentiment analysis § Extract information from text § Extracting structured information § JSON, Python classes, TypeScript types etc. § Via tools like JSON Schema or Open AI Functions Integration von Large Language Models in eigene Software APIs, Daten & Code Typical LLM scenarios: 16
  14. Extracting structured data (LangChain + Kor) Integration von Large Language

    Models in eigene Software APIs, Daten & Code DEMO 17
  15. Integration von Large Language Models in eigene Software APIs, Daten

    & Code Answering Questions on Data - Retrieval-augmented generation (RAG) Cleanup & Split Text Embedding Question Text Embedding Save Query Relevant Text Question Answer LLM 18 Vector DB Embedding model Embedding model 💡 Indexing / Embedding QA
  16. Learning about my company’s policies via Slack (LangChain) Integration von

    Large Language Models in eigene Software APIs, Daten & Code DEMO 19
  17. § Involve an LLM in making decisions (via ReAct –

    Reasoning & Acting) § Which actions to take (thought) § Taking that action (executed via your code) § Seeing an observation § Repeating until done § Tools to interact with the workflow from the LLM § LangChain tools and agents § Semantic Kernel native functions § OpenAI function calling § Reliable (structured) way to interact from the LLM with your code § Proprietary to OpenAI models (fine-tuned) Integration von Large Language Models in eigene Software APIs, Daten & Code Agents: Multi-step workflows 21
  18. Integration von Large Language Models in eigene Software APIs, Daten

    & Code Agents & ReAct – with OpenAI function calling LLM My code Query Some API Some database Prompt Function definitions Final answer Answer ❓ ❓ ❗ 💡 22 Function Function
  19. Ask for experts availability in my company systems (Whisper, node.js

    OpenAI SDK – Function Calling, TTS) Integration von Large Language Models in eigene Software APIs, Daten & Code DEMO 23
  20. Integration von Large Language Models in eigene Software APIs, Daten

    & Code 24 Angular PWA (&node.js Proxy) OpenAI Speech-to-Text TT “Panorama” (company-internal) OpenAI GPT-4 OpenAI Text-to-Speech Transcribe spoken text Transcribed text Check for experts availability with text Extract { experts, booking times } from text Structured JSON data Query Panorama API Generate response with experts availability Response Response with experts availability 🗣 🔉 Speech-to-text for response Response audio
  21. Integration von Large Language Models in eigene Software APIs, Daten

    & Code LLMs everywhere OpenAI-related (cloud) OpenAI Azure OpenAI Service Big cloud providers Google Model Garden on Vertex AI Amazon Bedrock Other providers Antrophic Cohere HuggingFace … Open-source Edge IoT Server Desktop Mobile Web Open-source 26
  22. § Open-source community drives innovation in Generative AI § HuggingFace

    is central place for it § Literally, every week a new and “better” LLM shows up 🤓 § Important factors § Use case § Parameter size § Quantization § Processing power needed § Mistral-based family shows big potential for local use cases (7B params) Integration von Large Language Models in eigene Software APIs, Daten & Code Open-source LLMs thrive 27
  23. Running Mistral OSS LLM locally (llama.cpp & LM Studio) Integration

    von Large Language Models in eigene Software APIs, Daten & Code DEMO 28
  24. § LLMs enable new scenarios & use cases to incorporate

    human language into software solutions § Fast moving and changing field § Every week something “big” happens in LLM space § Frameworks & ecosystem are evolving together with LLMs § Closed vs open LLMs § Competition drives invention & advancement § SISO (sh*t in, sh*t out) § Quality of results heavily depends on your data & input Integration von Large Language Models in eigene Software APIs, Daten & Code Current state 30
  25. Potential for LLM-powered human-machine workflows via universal interface agents Integration

    von Large Language Models in eigene Software APIs, Daten & Code Outlook 31
  26. § Semantic Kernel § https://learn.microsoft.com/en-us/semantic-kernel/overview/ § LangChain § https://www.langchain.com/ §

    LangChain Agents § https://python.langchain.com/docs/modules/agents/ § ReAct: Synergizing Reasoning and Acting in Language Models § https://react-lm.github.io/ § Prompt Engineering Guide § https://www.promptingguide.ai/ § OpenAI API reference § https://platform.openai.com/docs/api-reference § Azure OpenAI Service REST API reference § https://learn.microsoft.com/en-us/azure/ai-services/openai/reference § Hugging Face Inference Endpoints (for various OSS LLMs) § https://huggingface.co/docs/inference-endpoints/api_reference § Llama2 on Hugging Face § https://huggingface.co/blog/llama2 § OWASP Top 10 for LLM Applications § https://owasp.org/www-project-top-10-for-large-language-model-applications/assets/PDF/OWASP-Top-10-for-LLMs-2023-slides-v1_0_1.pdf Integration von Large Language Models in eigene Software APIs, Daten & Code Links 34