The Weight of Data: Rethinking Cloud-Native Systems for the Age of AI

The explosion of data and the rise of AI agents have brought cloud-native systems to a critical tipping point. While the CNCF has driven significant advancements, the challenges of fault tolerance and state management in the age of AI remain unresolved both at the core and the edge.

This keynote examines how data gravity, distributed environments, and the complexity of AI workloads demand a fundamental shift from stateless architectures to intelligent, state-aware platforms. This provides a unique opportunity for the CNCF and its ecosystem to pave the way for a new generation of cloud-native, resilient, cost efficient, scalable, and AI-ready systems, while tackling the pressing challenges that lie ahead.

Transcript

1. The Weight of Data Rethinking Cloud-Native Systems for the Age of AI Vincent Caldeira Holly Cummins Red Hat KubeCon & CloudNativeCon April 3, 2024
2. Title 2022
3. Title 2022
4. Title 2022
8. A new era of intelligent, data-driven systems tools smaller models orchestration model
9. A new era of intelligent, data-driven systems tools smaller models orchestration model agents
10. A new era of intelligent, data-driven systems tools smaller models orchestration model
11. but …
12. data gravity data nodes nodes
13. data gravity
14. data gravity
15. Cloud-native infrastructure was built for stateless microservices. AI is breaking that paradigm.
16. The question: How do we evolve Kubernetes to be AI-native?
17. Kubernetes already supports some stateful workloads.
18. Kubernetes already supports some stateful workloads.
19. Kubernetes already supports some stateful workloads.
20. Kubernetes already supports some stateful workloads.
21. CNCF projects have made databases work at scale.
22. CNCF projects have made databases work at scale. -Vitess
23. CNCF projects have made databases work at scale. -Vitess
24. CNCF projects have made databases work at scale. -Vitess
25. CNCF projects have made databases work at scale. -Vitess
26. CNCF projects have made databases work at scale. -Vitess -Rook
27. CNCF projects have made databases work at scale. -Vitess -Rook -K8s-Native Storage
28. Event-driven architectures (Knative, Kafka on K8s) enable streaming intelligence.
29. Event-driven architectures (Knative, Kafka on K8s) enable streaming intelligence.
30. But AI workloads go beyond traditional stateful applications.
31. AI agents don’t just store state
32. AI agents don’t just store state share —they
33. AI agents don’t just store state share, modify —they
34. AI agents don’t just store state share, modify, and react to data —they
35. AI agents don’t just store state share, modify, and react to data across distributed nodes. —they
36. Current PV and StatefulSet models require dynamic, multi-agent AI coordination.
37. Scaling state across nodes is limited—data locality, synchronization, and performance bottlenecks emerge at extreme scale.
38. Scaling state across nodes is limited—data locality, synchronization, and performance bottlenecks emerge at extreme scale.
39. Scaling state across nodes is limited—data locality, synchronization, and performance bottlenecks emerge at extreme scale.
40. Scaling state across nodes is limited—data locality, synchronization, and performance bottlenecks emerge at extreme scale.
41. We need: - AI-native scheduling - State management
42. NUMA-aware AI scheduling We need:
43. NUMA-aware AI scheduling We need:
44. GPU and topology-aware AI scheduling We need:
45. LLM Gateway
46. We need:
47. We need: Fault-tolerant recovery mechanisms to preserve AI state beyond pod restarts
48. CNCF projects that provide a foundation:
49. CNCF projects that provide a foundation: - Kueue
50. CNCF projects that provide a foundation: - Kueue - Envoy AI Gateway
51. CNCF projects that provide a foundation: - Kueue - Envoy AI Gateway - KServe + vLLM
52. CNCF projects that provide a foundation: - Kueue - Envoy AI Gateway - KServe + vLLM - Dapr + Dapr Agents
53. CNCF projects that provide a foundation: - Kueue - Envoy AI Gateway - KServe + vLLM - Dapr + Dapr Agents - OpenTelemetry
56. The CNCF has the foundation to lead the shift towards decentralised AI.
57. AI will change cloud-native. It will be intelligent, state-aware, and distributed. Let’s build it together.
58. Come see us at our booth Vaishnavi Natale More from Vincent: Green AI in Cloud Native Ecosystems: Strategies for Sustainability and Efficiency Vincent Caldeira & Tamar Eilam 15:15 Friday Level 1 | Hall Entrance S10 | Room A red.ht/KubeConEU-Keynote Live demo: “Build your own distributed cloud native AI agent in 20 minutes” 15:30 today, Demo Theater