KubeCon: To Infinity and Beyond: Seamless autoscaling with in-place resource resize for Kubernetes Pods

KubeCon+CloudNativeCon Europe 2024
https://kccnceu2024.sched.com/event/1YePO

1. Aya Ozawa, CloudNatix Kohei Ota, Apple To Infinity and Beyond: Seamless autoscaling with in-place resource resize for Kubernetes Pods
2. Self introduction Kohei Ota Senior Field Engineer at Apple CNCF Ambassador Chair of Cloud Native Community Japan Owner of SIG-Docs Japanese localization Twitter: @inductor__ GitHub: @inductor Aya Ozawa Member of Technical Staff at CloudNatix Co-organizer of Kubernetes Meetup Tokyo Twitter: @Ladicle GitHub: @Ladicle
3. Resource management is a key to smooth sailing!
4. Resource Requests & Limits Minimum guaranteed amount of a resource Maximum amount of a resource
5. Resource Requests & Limits Convert Requests & Limits to Container Config Check Requests Requests: cpu.share Limits: cpu.max, memory.max Pod Qos: oom_score_adj
6. Does the Pod fit the Node? Requests are used for scheduling CPU/Memory can be overcommitted
7. Degrade Performance Hit the Resource Limits! Memory Limits → OOM Killed CPU Limits → Throttled Overcommitted nodes could be in the same situation before reaching the limit.
8. Pod Quality of Service(QoS) Class & OOM Score For all containers At least one container For all containers OOM Killer oom_score_adj Node OOM!
9. Eviction Manager: Memory-pressure & Requests Evicted in order of Usage>Requests, Pod Priority(PP), Request-usage Gap
10. What goes wrong? Requests are too low Requests are too high Waste of cost Performance/Availability Degradation
11. What goes wrong? Requests are too low Requests are too high Waste of cost Performance/Availability Degradation If requests are not configured correctly, cluster-autoscaler cannot scale properly.
12. How do you set the right value? Observe Review Scale
13. Vertical Pod Autoscaler
14. What’s the VPA Scalable Objects e.g., Deployments, StatefulSets Observe Review Scale
15. Recommended Requests lowerBound target uncappedTarget upperBound Running with less resources is likely to damage performance/availability. Recommended amount of resources. Original target before capped by resource policy. Allocating beyond these resources is likely wasted. Keep the same ratio as the original requests and limits
16. Add samples to 24h half-decayed histograms How are targets estimated? 1. Fetch Metrics 2. 3. Find P90 values Newer samples are more important than older ones Default: OOM: Always: Peak usage in 5m Last usage Current usage CPU Memory Percentile
17. HPA vs. VPA VPA cannot be used with HPA on CPU or Memory Scenario where VPA and HPA do not work well together Use HPA on custom/external metrics (e.g., RPS) instead. Requests Per Seconds
18. 4 Update Modes (subject to change) Off Initial Recreate Auto Just calculate the recommended values Set resource only for pod creation Evict Pod and recreate it if resources not as recommended Equivalent to Recreate Only 1 1 & 3 All 1. Update 2. Evict 3. Set Resources VPA Components
19. Restarting is a serious disruption for some Pods (PDB) mitigates availability impact by Pod eviction! Successful scenario Non-optimal scenario High usage only at startup Resources not optimized Long-running Job e.g., ML workloads High running cost Pod Disruption Budget
20. In-place Pod Resizing Alpha
21. In-place Pod Resizing Alpha
22. Make resource settings mutable
23. Make resource settings mutable
24. Demo time! (wish us luck)
25. Demo time! (wish us luck) Succeeded? 👏👏👏 …Failed? It happens
26. Demo: Plan B (in case failure)
27. Detects a Pod that needs to be assigned to a Node Set to OCI spec Pass limits Call cgroups memory.max Setting resource limits by level
28. What has(had) to be implemented • API spec • Scheduler • Kubelet • Runtime ◦ cgroups v2 • Test
29. What has(had) to be implemented - API spec - Scheduler - Kubelet - Runtime - cgroups v2 - Test These are kubelet/api changes only! VPA implementation is another different story ☠
30. In-place resizing will be Beta soon (?) 🤷
31. VPA will support in-place mode • Still lots to consider in AEP(Autoscaler Enhancement Proposal): ◦ https://github.com/kubernetes/autoscaler/tree/master/vertical -pod-autoscaler/enhancements/4016-in-place-updates-supp ort ◦ Current implementation depends on runtime/kubelet decision on restarting containers, however in the proposed design, it should happen in VPA ◦ More use cases and feedback are wanted! ◦ We really need to be careful about memory scale down • More discussion in #sig-node-inplace-pod-resize on K8s Slack
32. Key Takeaways! • Resource management is important to managing the cluster efficiently and reliably. • VPA helps you to set the right amount of resources. • Current (GA) resource changes require restarting Pods ◦ There are some corner cases where there should not be • In-place resource resize solves this problem • VPA + in-place resource resize will bring you even more autonomous resource management on your cluster ◦ There are still LOTS of todos to make it happen
33. In-place resizing will make your voyage manageable
34. Please provide us feedback 🤞 https://sched.co/1YePO Kohei Ota Email: inductor@apple.com GitHub: @inductor Twitter: @inductor__ Aya Ozawa Email: aya@cloudnatix.com GitHub: @Ladicle Twitter: @Ladicle
35. Thank you!