CafeGPT: Serving LLMs Like Coffee With Kubernetes

Transcript

1. CafeGPT: Serving LLMs Like Coffee With Kubernetes Madhav Jivrajani & Kartik Ramesh Gopher credits: https://github.com/ashleymcnamara/gophers
2. Before we get into it, we’d like to start with what this talk isn’t! 2
3. This isn’t a deep dive into LLM inference and the internals of how it is enabled by Kubernetes. There’s much more suited talks at this conference and others on the topic! 3
4. We’ve been trying to educate ourselves on how many of these different pieces of the ecosystem fit together, and this is us hoping to share that picture with you. 4
5. Welcome to CafeGPT! ☕ 5
6. Welcome to CafeGPT! • CafeGPT is the best Cafe at KubeCon 2025. 6
7. Welcome to CafeGPT! • CafeGPT is the best Cafe at KubeCon 2025. • We have a new customer with a drink request. 7
8. Welcome to CafeGPT! • CafeGPT is the best Cafe at KubeCon 2025. • We have a new customer with a drink request. • There’s a manager and a barista to help fulfil the request! 8
9. Welcome to CafeGPT! • The manager directs the request to the barista. 9
10. Welcome to CafeGPT! • The manager directs the request to the barista. • The barista has access to coffee machines to help fulfill the request. 10
11. Welcome to CafeGPT! • The manager directs the request to the barista. • The barista has access to coffee machines to help fulfill the request. • Each coffee machine has many moving parts: the grinder, the steamer and the pressure valve. 11
12. Welcome to CafeGPT! • The barista uses a coffee machine to make the drink… 12
13. Welcome to CafeGPT! • The barista uses a coffee machine to make the drink… • … and finally serves the drink back to the customer via the manager 13
14. Welcome to CafeGPT! • Our customer loves the coffee! • They loved it because it was served in a timely manner and it’s not super expensive. 14
15. Its analogy time! 15
16. Serving In CafeGPT 16
17. • The drink request is the… request. Serving In CafeGPT 17
18. Serving In CafeGPT 18 • The drink request is the… request. • The manager is the router responsible for delegating the request to a barista.
19. Serving In CafeGPT 19 • The drink request is the… request. • The manager is the router responsible for delegating the request to a barista. • The barista is the inference engine - using resources available to it and converting requests to coffee.
20. Serving In CafeGPT 20 • The drink request is the… request. • The manager is the router responsible for delegating the request to a barista. • The barista is the inference engine - using resources available to it and converting requests to coffee. • The coffee machines are our GPUs that actually brew the responses!
21. Let’s continue makings things a little more concrete… what does the lifecycle of an LLM request look like? 21
22. LLM inference engines convert the user input into an output using a trained LLM on specialized hardware. Lifecycle of an LLM request
23. Lifecycle of an LLM request
24. Lifecycle of an LLM request
25. Lifecycle of an LLM request
26. Lifecycle of an LLM request
27. Lifecycle of an LLM request
28. Lifecycle of an LLM request
29. KV caching 1. Process the entire input sequence and store the KV Caches.
30. KV caching 2. Generate next token using the KV Cache
31. KV caching 2. Generate next token using the KV Cache until you hit <EOS>
32. KV caching speeds up LLM inference by 5x1, but it comes at an overhead of higher memory use, which becomes a bottleneck for concurrent processing of requests. KV caching
33. KV caching speeds up LLM inference by 5x1, but it comes at an overhead of higher memory use, which becomes a bottleneck for concurrent processing of requests. For Llama 3.1 8B, with 8K context length on an NVIDIA A100 GPU, you can only serve about 24 requests per second.2 KV caching Memory available 40GB Model Weights (FP16) 16GB KV Cache per request 1GB
34. KV caching speeds up LLM inference by 5x1, but it comes at an overhead of higher memory use, which becomes a bottleneck for concurrent processing of requests. For Llama 3.1 8B, with 8K context length on an NVIDIA A100 GPU, you can only serve about 24 requests per second.2 As a result, a lot of research has emerged to make better use of this bottleneck.3 KV caching Memory available 40GB Model Weights (FP16) 16GB KV Cache per request 1GB 1. https://magazine.sebastianraschka.com/p/coding-the-kv-cache-in-llms 2. https://lmcache.ai/kv_cache_calculator.html 3. https://arxiv.org/abs/2309.06180, https://lmcache.ai/tech_report.pdf
35. Prefill vs Decode Serving at CafeGPT has two phases: 1. Prefill - Do the prep work. 2. Decode - Multiple iterations of small units of work. Important workload characteristics: 1. Prefill is computationally intensive, but lasts a short duration. 2. Decode is memory intensive, and lasts a long duration. 3. You don’t know when the Decode will end.
36. Workload SLAs From a customer point of view, we want: 1. Low Time to Coffee <-> Request latency 2. Low Time between drops <-> Inter Token Latency 3. Low Time to first drop <-> Time to first token From a provider point of view we want: 1. More Coffee per second <-> Throughput 2. High machine utilization Different customers might have different SLAs.
37. Meet our novice barista
38. Meet our novice barista
39. Batch Brewing
40. Batching
41. Batching
42. Batching
43. Batching Throughput increases by over 10x due to batching.1 Request latency improves due to lower queue latencies.1 1. https://www.anyscale.com/blog/continuous-batching-llm-inference
44. Batching Interference between Prefill and Decode. Batch size controls iteration latency. Conflict between time to first drop and time between drops!
45. Disaggregation splits up prefill and decode processing across different instances. Prefill / Decode Disaggregation
46. That’s pretty cool, but how do I deploy and run this? 46
47. Same Infra, New Workload?
48. Same Infra, New Workload? • Kubernetes has become the defacto choice for a large percentage of companies to build their platforms on.
49. Same Infra, New Workload? • Kubernetes has become the defacto choice for a large percentage of companies to build their platforms on. • Inference is an interesting new workload with unique characteristics that can be served very well* without having to reinvent the wheel.
50. Same Infra, New Workload? • Kubernetes has become the defacto choice for a large percentage of companies to build their platforms on. • Inference is an interesting new workload with unique characteristics that can be served very well* without having to reinvent the wheel. • In fact, the community has been relentlessly evolving the core and ecosystem projects to better support this workload.
51. What do these efforts look like? Let’s build bottom up.
52. Let’s say you want to host a model and serve requests.
53. Espresso (Smol) Models Maybe your model is small enough to need just one GPU. ... resources: limits: nvidia.com/gpu: 1
54. Espresso (Smol) Models Maybe… you just need a fraction of a GPU. ... resources: limits: nvidia.com/mig-1g.5gb.shared: 1
55. Maximising Hardware Efficiency As a vendor, you may statically pre-partition your devices and expose those partitions.
56. Maximising Hardware Efficiency However, this may not maximise hardware efficiency. For example, if I need 1/8th of an H100, I might be forced to run on a slice larger than what I need - leading to wastage.
57. Maximising Hardware Efficiency I also may not need a particular type of GPU - for example, all I may care about is 20Gi of GPU memory, I don’t care where it comes from. ... resources: limits: nvidia.com/gpu: 1
58. Maximising Hardware Efficiency DRA (Dynamic Resource Allocation) can help with that!
59. Maximising Hardware Efficiency DRA (Dynamic Resource Allocation) can help with that! For example, vendors may be able to partition devices on the fly (KEP-4815): ... requests: ... "device.attributes['gpu.nvidia.com'].profile == '1g.5gb'”
60. Maximising Hardware Efficiency DRA (Dynamic Resource Allocation) can help with that! ... requests: ... "device.capacity['nvidia.com']. memory.compareTo(quantity('10Gi')) >= 0"”
61. Now that we’ve made sure we’re getting the most bang for our buck…
62. Seems Familiar? We want our cafe to be able to handle many different kinds of scenarios. 1. Surges in the morning and lunch, slows at night. 2. Surges of specific drinks on occasions 3. Different workloads For LLM serving, these changes manifest as conversation heavy vs coding heavy tasks, or increased requests for certain models.
63. Let’s use Kubernetes HPA to scale our serving system! CPU / memory utilization ❌ Horizontal Pod Autoscaler
64. Concurrency: ❌ Summarization: 1000 input, 100 output Code generation: 100 input, 1000 output Traditional autoscaling metrics are not a great fit for LLM workloads. Horizontal Pod Autoscaler
65. An increasing queue length indicates increasing that our barista can’t keep up with their orders. New requests will face blocking, until pending requests are completed (unknown) Queue Lengths
66. An increasing queue length indicates increasing that our barista can’t keep up with their orders. New requests will face blocking, until all pending requests are completed (unknown) Autoscaling on queue lengths boosts your processing throughput and reduce queuing delays. However, it does not lower latency of your processing requests. Queue Lengths
67. Batch Sizes If latency is a concern, keep batch sizes low and monitor the number of active batched tokens. Scaling on batch size, can help you scale earlier than queue buildup and still meet SLAs, but you might over-react to temporary spikes.
68. For improving latency, scale up your Disaggregated Prefill deployments. Two options for scaling: 1. Scale up P:D instances keeping the ratio constant a. Good for increased requests for constant workload 2. Scale up individual P or D instances a. Good if you don’t have a lot of GPUs, or if you workload has shifted. Prefill Decode Disaggregation
69. KV cache utilization Often, KV cache space can be the main bottleneck. 1. Reactive: Monitor Scale if KV cache utilization exceeds a threshold. 2. Proactive: Use your workload to estimate how much KV cache you need to serve requests while meeting SLAs Memory available 40GB Model Weights (FP16) 16GB KV Cache per request 1GB
70. Seems Familiar? With autoscaling in place: • How do we load balance across these replicas?
71. Seems Familiar? With autoscaling in place: • How do we load balance across these replicas? • Can we do better than round-robin?
72. Seems Familiar? With autoscaling in place: • How do we load balance across these replicas? • Can we do better than round-robin? ◦ Yes! Sending requests round robin might actually result in degraded performance. ◦ You may end up with hotspots because each workload is not the same.
73. Good Routing and Better SLAs • Your inference engine is caching processed tokens (KV Cache).
74. Good Routing and Better SLAs • Your inference engine is caching processed tokens (KV Cache). • The router can take this information to better balance requests. https://github.com/kubernetes-sigs/gateway-api-inference-extension
75. Good Routing and Better SLAs • You don’t want to route to the same instance all the time because you might overload it. • You might also want to route to another instance as an escape hatch. But which one?
76. Good Routing and Better SLAs • You don’t want to route to the same instance all the time because that might become the bottleneck then. • You might also want to route to another instance as an escape hatch. • We can route to the least loaded replica! ◦ KV cache utilization ◦ Queue lengths ◦ Some custom metric that matters to you
77. Good Routing and Better SLAs llm-d and AIBrix are two such ecosystem projects that help with better routing and load balancing. https://llm-d.ai/, https://aibrix.readthedocs.io/latest/
78. Phew! That was a lot of info. Let’s zoom out and look at the picture we’ve been constructing. 78
79. In Conclusion
80. In Conclusion
81. In Conclusion
82. In Conclusion
83. So, what’s the answer?
84. So, what’s the answer?
85. We hope we’ve convinced you that you don’t need to be proficient in language modelling and that when you try to work with these systems you can ground yourself in the fact that serving LLMs is as tractable as serving coffee!
86. Thank you for visiting CafeGPT! 🥰 86
87. 87 Feedback & Tips Welcome!