[2023] Complexities You Should Care about Doing Real-time ML

If you are a data scientist or a platform engineer, you probably can relate to the pains of working with the current explosive growth of Data/ML technologies and toolings. With many overlapping options and steep learning curves for each, it’s increasingly challenging for data science teams. Many platform teams started thinking about building an abstracted ML platform layer to support generalized ML use cases. But there are many complexities involved, especially when dealing with Real-time or Near Real-time ML.

In this talk, we’ll discuss why ML platforms can benefit from a simple and “invisible” abstraction. We’ll offer some evidence on why you should consider leveraging streaming technologies despite all the hard challenges you have heard. We’ll share learnings (combining both ML and Infra perspectives) about some of the hard complexities involved in building such simple abstractions, the design principles behind them, and some counterintuitive decisions you may come across along the way.

By the end of the talk, I hope data scientists can walk away with tips to be more productive with ML workflows, and platform engineers learned a few architectural and design tricks that can make your organization future-proof.

Transcript

1. Invisible Interfaces Zhenzhong Xu Cofounder & CTO @ claypot.ai July, 2023 Considerations for Abstracting Complexities of a Real-time ML Platform
2. The discovery of something invisible
3. The Invisible Interface Ubiquitous Easy and responsive Just works! The endeavor to make things useful
4. Real-time Decisions that powers your business Fraud prevention Personalization Customer support Dynamic pricing/discounting Trending products Risk Assessment Account Take Over Ads ETA Network analysis Sentiment analysis Object detection …
5. The world is moving towards real-time • Instacart: The Journey to Real-Time Machine Learning (2022) ◦ Directly reduces millions of fraud-related costs annually. • LinkedIn’s Real-time Anti-abuse (2022) ◦ LinkedIn moved from an offline pipeline (hours) to real-time pipeline (minutes), and saw 30% increase in bad actors caught online and 21% improvement in fake account detection. • How WhatsApp catches and fights abuse (2022 | slides) ◦ A few 100ms delay can increase the spam by 20-30%. • How Pinterest Leverages Realtime User Actions in Recommendation to Boost Engagement (2022) ◦ According to Pinterest, this “has been one of our most impactful innovations recently, increasing Home feed engagement by 11% while reducing Pinner hide volume by 10%.” • Airbnb: Real-time Personalization using Embeddings for Search Ranking (2018) ◦ Moving from offline scoring to online scoring grows bookings by +5.1% 5
6. Real-time Decisions Data Fabric for Real-time AI Data Infrastructure Exploration & Research Model Architecture & Turning Model Analysis & Selection Ingestion & Transport Security & Governance Multi-tenancy Isolation Data Sources Storage Query & Compute LLM Prompt Engineering Workflow Orchestration Analytics / Visualization
7. Model Serving Model Training Model Monitoring Model Evaluation Prediction Input Training Input Data Monitoring Data Model Flow Data Flow Product Ecosystem Analytics ecosystem
8. The hard things towards real-time decisions • Data silo and staleness • Collaboration overhead • Tech complexity
10. Challenge 1: From Experimentation to Production • Slow prototyping • Local vs. remote execution • Divergent language & runtime
11. Local Experimentation with Traditional Models
12. Local Experimentation with LLMs
14. Sources Feature store online + offline Prediction service Feature API Create, experiment, & deploy features Computation engines Training service Feature catalog Data scientists Central repo
15. Local/Single Machine Remote/Distributed Need an invisible interface to plug into compute ecosystems
16. Declare features with familiar APIs @transformation def average_transaction_amount_by_merchant( tx: Transactions, wspec: WindowSpec): return tx.groupby(["cc_num", "merchant"])["amt"].window(wspec).mean()
17. 17 Workload Compiler / Optimizer Deployment Relational Expression @transformation def transaction_count(tx: Transactions, wspec: WindowSpec): return tx[tx.status == "failed"].groupby("account_id").window(wspec).count() Data Science Friendly: Python <> SQL
18. Workload Compiler/Optimizer Deployment Relational Expression @transformation def transaction_count(tx: Transactions, wspec: WindowSpec): return tx[tx.status == "failed"].groupby("account_id").window(wspec).count() Same code can run on different computation engines Compile into a relational expression (RE), which is SQL equivalent Intermediate Representation Compile & optimize RE into the computation engine (e.g., Panda, DuckDb, Flink, Spark) best suited for the job Spin up and manage computation jobs
19. Solution 1: Relational Expression based Compilation • Unified yet familiar API • Pluggable to many compute engines • Minimize human error • Prototype in minutes
20. Challenge 2: Streaming and Batch Divided • Evolving architecture • Difficult to backfill • Train-predict inconsistencies
21. Data Source In-motion Compute At-rest Compute Online Storage Offline Storage Online Query (serving) Mixed Query (backfill) Offline Query (training) Lambda Architecture
22. Kappa (Streaming) Architecture Data Source In-motion Compute (Backfill from historical log) Materialized Views Online Query (serving) Offline Query (training) batch transformation streaming transformation
23. Unified Architecture Data Source In-motion Compute (intelligent backfill from dual sources) Materialized Views Online Query (serving) Offline Query (training) batch transformation streaming transformation DWH backed logs Backing
24. Batch and streaming source unified to simplify backfill Time DWH Stream Dual source cutover
25. Streaming Leaning Batch Leaning Need an invisible interface to plug into storage ecosystems
26. Data Fabric for a Streaming Pipeline
27. Data Fabric for a Unified Backfill Pipeline
28. Training dataset backfill requires point-in-time correctness Time Feature data Feature data Feature data Prediction events Feature data
29. Point-in-time joins to generate training data 29 Proprietary & Confidential Given a spine (entity keys + timestamp + label), join features to generate training data spine_df train_df = pitc_join_features( spine_df, features=[ "tx_max_1h", "user_unique_ip_30d", ], ) inference_ts tid cc_num user_id is_fraud 21:30 0122 2 1 0 21:40 0298 4 1 0 21:55 7539 6 3 1 inference_ts tid cc_num user_id is_fraud tx_max_1h user_unique_ip_30d 21:30 0122 2 1 1 … … 21:40 0298 4 1 1 … … 21:55 7539 6 3 3 … … ts cc_num tx_max_1h 9:20 2 … 10:24 2 … 20:00 4 … cc_num_tx_max_1h ts user_id unique_ip_30d 6:00 1 … 6:00 3 … 6:00 5 … user_unique_id_30d
30. Solution 2: Abstract streaming and batch data storage • Unified streaming & batch source • Unified online & offline feature stores • Pluggable to most storage technologies
31. Challenge 3: It should just work! • Cost, latency, correctness surprises! • Lack optimizations knobs
32. Batch processing (cheap and correct) Cost Latency Correctness Stream processing without consistency (fast and cheap) Stream processing with consistency enforced (fast and correct)
33. Workload Compilation Optimization Relational Expression @transformation def transaction_count(tx: Transactions, wspec: WindowSpec): return tx[tx.status == "failed"].groupby("account_id").window(wspec).count() Optimization Various intelligent optimization can be done to make appropriate tradeoff across storage and compute systems. Deployment
34. Customer managed in your own cloud Guardrail for schema changes Tunable workload optimization Claypot Feature SDK (Python) Feature Catalog Online store Offline store Feature Serving Filter Scan Scan Union Join Unified Processing Filter
35. Solution 3: Optimization knobs • Abstract optimization complexity • User controls with high level knobs • Trust, no surprises!
36. Make invisible interface possible! • Ubiquitous • Easy and responsive • Just works! https://zhenzhongxu.com/ zhenzhong@claypot.ai the invisible interface