Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Building and deploying LLM applications with Ap...

Building and deploying LLM applications with Apache Airflow

Behind the growing interest in Generate AI and LLM-based enterprise applications lies an expanded set of requirements for data integrations and ML orchestration. Enterprises want to use proprietary data to power LLM-based applications that create new business value, but they face challenges in moving beyond experimentation. The pipelines that power these models need to run reliably at scale, bringing together data from many sources and reacting continuously to changing conditions.

This talk focuses on the design patterns for using Apache Airflow to support LLM applications created using private enterprise data. We’ll go through a real-world example of what this looks like, as well as a proposal to improve Airflow and to add additional Airflow Providers to make it easier to interact with LLMs such as the ones from OpenAI (such as GPT4) and the ones on HuggingFace, while working with both structured and unstructured data.

https://odsc.com/speakers/building-and-deploying-llm-applications-with-apache-airflow/

Kaxil Naik

October 31, 2024
Tweet

More Decks by Kaxil Naik

Other Decks in Technology

Transcript

  1. Kaxil Naik Apache Airflow Committer & PMC Member Senior Director

    of Engineering @ Astronomer @kaxil @kaxil @kaxil
  2. Chat about Chatbots IRL Business Leader: Shouldn’t we be doing

    something with LLMs? Intern: Hey I built this notebook with LangChain? Data Engineer: Okay, yeah let me put that into production.
  3. ▪ Ingestion from several sources ▪ Day 2 operations on

    data pipelines: changing data sources, network blips etc ▪ Data preparation: data cleaning & transformation ▪ Data privacy: redacting PII data, tracking data lineage for audits ▪ Data freshness: timeliness, SLAs ▪ Model deployment & monitoring ▪ Experimentation & fine-tuning: different models, LLMs, SLMs etc ▪ Feedback Loops Going from “Idea to Production” with LLM Apps involves solving a lot of data engineering problems:
  4. Source: https://python.langchain.com/docs/use_cases/question_answering/ Typical Architecture for Q&A use-case using LLM -

    Airflow for RAG Data Ingestion & Processing Retrieval Output Storage Splitting Document Loading Vectorstore Database PDFs URLs LLM <Answer> Prompt Splits Relevant Splits Query <Question> Retrieval Augmented Generation (RAG)
  5. Python Native The language of data scientists and ML engineers.

    Pluggable Compute GPUs, Kubernetes, EC2, VMs etc. Common Interface Between Data Engineering, Data Science, ML Engineering and Operations. Data Agnostic But data aware. Extensible Standardize custom operators and templates for common DS tasks across the organization. Monitoring & Alerting Built in features for logging, monitoring and alerting to external systems. Ingestion Extract and load data into vectordbs and other destinations Day 2 Ops Handle retries, dependencies, and all other day 2 ops associated with data pipelines Airflow is a Natural Fit… Document Parsing Decorator and pythonic interfaces for standard LLM tools
  6. Problem Statement: We have customers, employees, and community members that

    ask questions about our product (Astro) and Airflow with answers that exist across several sources of documentation. How do we provide an easy interface for folks to get their questions answered without adding further strain to the team and Airflow Contributors?
  7. ▪ Airflow gives a framework to load data from APIs

    & other sources into LangChain ▪ LangChain helps pre-process and split documents into smaller chunks depending on content type ▪ After content is split into chunks, each chunk is embedded into vectors (semantic representations) ▪ Those vectors are written to Weaviate for later retrieval Data Ingestion, Processing, and Embedding Embed chunks Write to Weaviate Pre-process and split into chunks 🦜🔗 LangChain Docs (.md) files Slack Messages GitHub issues Docs (.md) files
  8. [ 0.13450, 0.72421, 0.20943, 0.18699, 0.75932, 0.69794 Vector Embedding for

    Unstructured Data “Apache Airflow is a platform created by the community to programmaticall y author, schedule and monitor workflows.ˮ Chunk of unstructured text Embedding Model Vector
  9. Users can interact with UI or Slack Bot; they both

    use the same API ▪ Original prompt gets reworded 3x using gpt-3.5-turbo ▪ Answer is generated by combining docs from each prompt and making a gpt-4 call ▪ State is stored in Firestore and prompt tracing is done through LangSmith 🦜🔗LangChain User Asks a Question Web App Slack Bot Original Prompt Rewording 2 Rewording 1 Rewording 3 Reword to get more related documents Vector DB search with prompts Combine docs and make final LLM call to answer 🦜 🔗 Prompt Orchestration and Answering
  10. [ 0.13450, 0.72421, 0.20943, 0.18699, 0.75932, 0.69794 Using Embeddings Vector

    A [ 0.17450, 0.22621, 0.10643, 0.18699, 0.55932, 0.99794 Vector B Question Related Document
  11. ▪ Airflow DAGs process feedback async to evaluate answers on

    helpfulness,, relevance, and publicness ▪ If answer is good, it gets stored in Weaviate and can be used as a source for future questions ▪ UI also shows the most recent good prompts on the homepage When a user submits feedback, it gets stored in Firestore and LangSmith for later use User Rates Answer 🦜 🔗 Fetch new runs: input, output, and user feedback Classify Q&A according to helpfulness, relevance, and public 🦜🔗 LangChain If good answer, write to vector DB to use in future answers If good answer, mark as good to show on Ask Astro homepage On schedule LLM & Product Feedback Loops
  12. Running this in production meant: ▪ Experimenting with different sources

    of data to ingest ▪ Running the pipelines on a schedule and ad-hoc (new Airflow release) ▪ Running the same workloads with variable chunking strategies ▪ Needing to retry tasks due to finicky python libraries and unreliable external services ▪ Giving different parts of the workload variable compute ▪ Creating standard interfaces to interact with external systems
  13. ▪ Experimenting with different sources of data to ingest ▪

    Running the pipelines on a schedule and ad-hoc (new Airflow release) ▪ Running the same workloads with variable chunking strategies ▪ Needing to retry tasks due to finicky python libraries and unreliable external services ▪ Giving different parts of the workload variable compute ▪ Creating standard interfaces to interact with external systems Running this in production meant: Which is what Airflow’s great at!
  14. a16z’s Emerging LLM App Stack Orchestration (Python/DIY, LangChain, LlamaIndex, ChatGPT)

    APIs/Plugins (Serp, Wolfram, Zapier, etc.) App Hosting (Vercel, Steamship, Streamlit, Modal) Query Output Prompt Few-shot examples Contextual data Playground (OpenAI, nat.dev, Humanloop) Data Pipelines (Databricks, Airflow, Unstructured, etc.) Embedding Model (OpenAI, Cohere, Hugging Face) Vector Database (Pinecone, Weaviate, Chroma, pgvector) LLM Cache (Redis, SQLite, GPTCache) Logging/LLMops (Weights & Biases, MLflow, PromptLayer, Helicone) Validation (Guardrails, Rebuff, Guidance, LMQL) Proprietary API (OpenAI, Anthropic) Open API (Hugging Face, Replicate) Opinionated Cloud (Databricks, Anyscale, Mosaic, Modal, Runpod) Cloud Provider (AWS, GCP, Azure, Coreweave) LLM APIs and Hosting Gray boxes show key components of the stack, with leading tools / systems listed. Arrows show the flow of data through the stack. Contextual data provided by app developers to condition LLM outputs Prompts and few-shot examples that are sent to the LLM Queries submitted by users Output returned to users Legend
  15. AskAstro has a few parts of this… Orchestration (Python/DIY, LangChain,

    LlamaIndex, ChatGPT) APIs/Plugins (Serp, Wolfram, Zapier, etc.) App Hosting (Vercel, Steamship, Streamlit, Modal) Query Output Prompt Few-shot examples Contextual data Playground (OpenAI, nat.dev, Humanloop) Data Pipelines (Databricks, Airflow, Unstructured, etc.) Embedding Model (OpenAI, Cohere, Hugging Face) Vector Database (Pinecone, Weaviate, Chroma, pgvector) LLM Cache (Redis, SQLite, GPTCache) Logging/LLMops (Weights & Biases, MLflow, PromptLayer, Helicone) Validation (Guardrails, Rebuff, Guidance, LMQL) Proprietary API (OpenAI, Anthropic) Open API (Hugging Face, Replicate) Opinionated Cloud (Databricks, Anyscale, Mosaic, Modal, Runpod) Cloud Provider (AWS, GCP, Azure, Coreweave) LLM APIs and Hosting Gray boxes show key components of the stack, with leading tools / systems listed. Arrows show the flow of data through the stack. Contextual data provided by app developers to condition LLM outputs Prompts and few-shot examples that are sent to the LLM Queries submitted by users Output returned to users Legend
  16. Airflow is foundational to best practices for all of this.

    Data Governance ▪ How do you account for private data? ▪ How do you provide transparency into data lineage? Fine Tuning ▪ Does it improve results? ▪ How much does it cost? Feedback Loops ▪ Semantic cache for correct responses ▪ Ranking sources based on accuracy and ranking accordingly ▪ Prompt clustering – what are people asking? …but there’s even more to consider.