Build Real-Time Next Generation AI Applications with Kafka and Flink by Jeffrey Lam

Speaker : Jeffrey Lam @Confluent
Bio : Jeffrey Lam is a Staff Solutions Engineer at Confluent. He is a big data specialist with over 20 years of experience in pre-sales and enterprise architecture, having worked for global technology leaders such as Confluent, Splunk, ServiceNow, IBM, and Oracle.

In this event, we will explore how to build next-generation real-time AI applications using Kafka and Flink. Jeffrey Lam will share his expertise in this field and discuss how to continuously enrich trustworthy data streams to quickly build, deploy, secure, and scale real-time AI applications in the era of booming artificial intelligence.

Participants will gain a deep understanding of how to use Kafka and Flink as a foundation for processing real-time and batch data. You will learn how these technologies can become a core competitive advantage for companies and their wide adoption in both commercial and open-source communities. Additionally, you will gain valuable insights into building scalable and secure real-time AI applications.

Transcript

1. Building Real-time Next Generation AI Applications with Kafka and Flink Jeffrey Lam Staff Solutions Engineer, Confluent Inc.
2. “Our latest research estimates that generative AI could add the equivalent of $2.6 trillion to $4.4 trillion annually across the 63 use cases we analyzed.” Source: Economic Potential of Generative AI, McKinsey
3. Generative AI is a revolutionary tool… …and it’s only getting better. July 2022 July 2023 Source: https://twitter.com/nickfloats/status/1676279157620199424?s=46&t=plcKoQYXnokFvxs3ieVg3Q
4. Vectors man [0.243, 0.765, …] woman [0.293, 0.774, …] Similar Vectors plotted in space are placed near one another
5. Generative AI: the hottest topic in tech… …but what makes it different? - AI models that generate content (e.g., text, pictures) rather than make predictions. - Uses Foundation Models (e.g., LLMs) that are prohibitively expensive ($100M+) to train. 175B neurons. - Models are trained on 1+ year-old public data. - However, models are inherently reusable.
6. LLMs can drive value for your business… …but only if they have context from your data.
7. Without contextualized, trusted, current data LLMs can’t drive meaningful value Source: https://www.wired.com/story/air-canada-chatbot-refund-policy/
8. How to create the next generation AI applications? Whatever your AI use case, the recency, quality, trustworthiness and instant applicability of data is as important as the models themselves. Company Data Trusted Data Realtime Data
9. Without context, trustworthiness or real-time data applicability, LLMs can’t drive meaningful value What is the status of my flight to New York? It is currently delayed by 2 hours and expected to depart at 5 pm GMT. Is there another flight available to the same city that will depart and arrive sooner? What are the seating options and cost? The next available flight to New York with United departs later but will arrive faster than your current flight. The only available seats in this flight are first class window seats and costs $1,500. Can your GenAI assistant remember data from an earlier conversation? What is the source of this information? Is this trustworthy? Is it fresh and accurate? How do you securely augment customer data with real-time data and process them on the fly to provide meaningful insights?
10. Retrieval Augmented Generation 10 Retrieval-augmented generation (RAG) is a technique for enhancing the accuracy and reliability of generative AI models with facts fetched from external sources. LLM APP Databases Vector Embeddings Vector Database Users expect very low latency as the application must traverse multiple data sources before communicating with the LLM. Microservices
11. The reality of today’s data integration strategy A giant mess of monolithic point-to-point connections with data fidelity and governance challenges
12. Traditional enterprise data architecture is a AI innovation bottleneck Historic Public Data Intelligent Business-Specific Co-Pilot User Interaction ?? Enterprise data architecture In-context learning & prompt-time assembly Generative AI Model
13. A high level picture Build a real-time, contextual and trustworthy knowledge base for your AI applications CONNECT PROCESS GOVERN SHARE Real-time AI Apps Data Systems STREAM Pricing Inventory Payments Personalization Fraud Supply Chain Recommendations From Data Mess To Data Products To Instant Value Everywhere Kafka Flink
14. AI enabled stream enrichment in Apache Flink INSERT INTO enriched_reviews SELECT id , review , invoke_gpt4(prompt,review) as score FROM product_reviews ; 14 The Prompt “Score the following text on a scale of 1 to 5 where 1 is negative and 5 is positive returning only the number” Reviews 1, “This was the worst decision ever” 2, “Not bad. Could have been cheaper” 3, “Amazing! Game Changer!” Output 1, “This was the worst decision ever”, 1 2, “Not bad. Could have been cheaper”, 3 3, “Amazing! Game Changer!”, 5
15. Improving product descriptions using GenAI Description CETAPHIL BODY MOISTURIZING CREAM FOR DRY TO VERY DRY SKIN: Instantly replenishes, intensely nourishes and soothes skin dryness for 48 hours CLINICALLY PROVEN TO RESTORE SKIN'S MOISTURE BARRIER IN 1 WEEK. Binds water to the skin, preventing moisture loss to hydrate and protect skin from dryness NEW AND IMPROVED INGREDIENT BLEND: Now with hydrating glycerin and skin essential vitamins B5 (panthenol) and B3 (niacinamide) DEVELOPED FOR EVEN THE MOST SENSITIVE SKIN: The hypoallergenic, non-comedogenic formula is free of fragrances, parabens and sulfates DERMATOLOGIST RECOMMENDED for Sensitive Skin 15 Prompt Take the following product description and summarize the product in less than 10 words. Precede the summary with the word summary. Choose one word from the following words to categorize the product. Cream, Skincare, Beauty, Healthcare. Precede the category with the word category. Code SELECT name as Name , REGEXP_EXTRACT(a.response,".*Category:(.*),0") as Category , REGEXP_EXTRACT(a.response,"Summary:(.*),0") as Summary FROM ( SELECT name , invoke_gpt4(prompt, description) response FROM products ) a ; Output | Name | Category | Summary | | -------- | --------- | –---------------------------------------------------------- | | Cetaphil | Skincare | Moisturizing cream for dry skin, restores moisture barrier. | Summary: Moisturizing cream for dry skin, restores moisture barrier. Category: Skincare. GPT
16. 0 50,000 100,000 150,000 2020 2021 2022 2016 2017 2018 Flink Kafka Kafka & Flink - 2 of Top 5 Apache Projects that goes hand-in-hand together for event stream processing >75% of the Fortune 500 estimated to be using Kafka >100,000+ orgs using Kafka >41,000 Kafka meetup attendees >750 Kafka Improvement Proposals >12,000 Jiras for Apache Kafka Two Apache Projects, Born a Few Years Apart Monthly Unique Users
17. 17 Kafka A Distributed Commit Log. Publish and subscribe to streams of records. Highly scalable, high throughput. Supports transactions. Persisted data. Reads are a single seek & scan Writes are append only Apache Kafka - Publish & Subscribe Reimagined 1. Publish Stream Events 2. Store 3. Process & Consume
18. Apache Flink Write microservices to process your data in real-time Kafka Connect API Reliable and scalable integration of Kafka with other systems – no coding required. Orders Customers Flink Apache Kafka and Microservices Table Microservices 18
19. Filters INSERT INTO STREAM high_readings SELECT sensor, reading, FROM readings WHERE reading > 41 ** The above query runs indefinitely and produces the resulting in a new topic “high readings”. Flink Kafka Kafka
20. Joins INSERT INTO enriched_readings SELECT senior, reading, area, brand_name FROM readings INNER JOIN brands b ON b.sensor = readings.sensor Kafka Flink Kafka
21. Aggregate INSERT INTO avg_readings SELECT sensor, AVG(reading) AS location FROM readings GROUP BY sensor Flink Kafka Kafka
22. Core processing systems External data Unstructured data Systems of Record Browser mobile Telemetry DWH, Data Lake SaaS apps … Infrastructure Data Sources How to do it with Kafka & Flink Event-driven Decoupled architecture Immutable Robust security controls Real-time and performant Fully managed cloud-native service Vector Databases Model Building / Fine-tuning GenAI Consumer & Gateway GenAI Agents in SaaS Cluster 120+ pre-built Connectors Stream Processing Stream Governance Create a real-time knowledge base Bring real-time context at query time Build governed, secured, and trusted AI Experiment, scale and innovate faster +
23. LLM-enabled applications have four steps Data Augmentation Prepare data for a real-time knowledge base and contextualization in LLM queries Inference Programmatically connecting relevant information with each prompt Workflows Parsing natural language to synthesize necessary information and apply contextual reasoning on-the-fly Post-Processing Validates outputs and enforces business logic to detect hallucinations and ensure trustworthy responses
24. How Kafka + Flink Work: Data Augmentation Prepare data for a real-time knowledge base and contextualization in LLM queries Schema Registry Kafka Consumer Group Kafka Sink Connector or Native Integration Kafka Sink Connector or Native Integration Operational Data Store (Unstructured Data) Vector Store Versioned Data Products in git Embedding API Gateway Embedding Model Instances Vector Embedding Service Schema Specs Terraform Flink
25. How Kafka + Flink Work: Inference Programmatically connecting relevant information with each prompt Schema Registry Confluent Cloud Consumer Group Versioned Data Products in git LLM API Gateway LLM Instances LLM Service Schema Specs Terraform Web Application Vector Store Flink
26. How Kafka + Flink Work: Workflows Parsing natural language to synthesize necessary information and apply contextual reasoning Schema Registry Confluent Cloud Consumer Group Versioned Data Products in git LLM API Gateway LLM Instances LLM Service Schema Specs Terraform Web Application Vector Store Operational Data Store (RDBMS) Reasoning Agent Flink
27. Schema Registry Confluent Cloud Consumer Group Versioned Data Products in git LLM API Gateway LLM Instances LLM Service Schema Specs Terraform Web Application Vector Store Operational Data Store (RDBMS) Reasoning Agent How Kafka + Flink Work: Post-Processing Enforce Business Logic and Compliance Requirements with LLM Outputs Post- Processing Consumer Group Flink
28. Further Materials 28 Github and Video https://github.com/brittonlaroche/Confluent-Kafka-Vector-Encoding?tab=readme-ov-file
29. Real-time Weather and Flight Status Embeddings into Knowledge Database 29 Building Real-Time, Intelligent AI Copilots https://techcommunity.microsoft.com/t5/ai-azure-ai-services-blog/building-real-ti me-intelligent-ai-copilots-with-confluent-cloud/ba-p/3932183
30. How to get started with Kafka and Flink 30 https://developer.confluent.io
31. 31 cnfl.io/ask-the-community Ask questions, share knowledge and chat with your fellow community members! Join your local Kafka User Group! meetup.com/taipei-kafka/ Learn Apache Kafka® with Confluent
32. Confluent
33. THE AI FRAUD DETECTION CHALLENGE: Looking at the series of events surrounding a transaction to derive context in a timely manner