A JVM Threading Model for the containerized times

Presented in the Strange Loop event in 2023.

Transcript

1. A JVM threading model for the containerized times Flavio Brasil, Principal Engineer Luiz Hespanha, Principal Engineer Systems Performance @ Nubank { } ...
2. Nubank
3. Nubank Hespanha
4. The perfect storm 01 { } .. ..
5. PIX Running since the end of 2020, Pix is an instant payment platform created and managed by the monetary authority of Brazil, the Central Bank of Brazil (BCB), which enables the quick execution(max 10 seconds) of payments and transfers 24/7.
6. PIX Monthly transfers (thousands) - 4 Billion in July!
7. PIX Essential for people's day-to-day
8. Payday madness Nubank's PIX down today? Service experiencing instability Number of failures (RED) increasing
9. Understanding the problem 02 { } .. ..
10. The crash resolution paradox
11. The system normally operates at a low CPU usage The crash resolution paradox
12. The system normally operates at a low CPU usage But when there's a load spike, the CPU becomes a bottleneck The crash resolution paradox Flavio
13. The crash resolution paradox The system normally operates at a low CPU usage But when there's a load spike, the CPU becomes a bottleneck Latencies skyrocket, system sometimes become unresponsive
14. The crash resolution paradox The system normally operates at a low CPU usage But when there's a load spike, the CPU becomes a bottleneck Latencies skyrocket, system sometimes become unresponsive Resolution: more CPU capacity!?
15. Cluster-wide instability
16. Some crashes escalated to k8s nodes becoming saturated Cluster-wide instability
17. Some crashes escalated to k8s nodes becoming saturated A few systems consumed all CPU resources and became noisy neighbors Cluster-wide instability
18. Cluster-wide instability Some crashes escalated to k8s nodes becoming saturated A few systems consumed all CPU resources and became noisy neighbors Several nodes become saturated and instability spread to collocated services
19. Symptoms of a bottleneck Flavio
20. As described by the Universal Scalability Law (USL), efficiency can drop significantly when a bottleneck is reached Symptoms of a bottleneck
21. As described by the Universal Scalability Law (USL), efficiency can drop significantly when a bottleneck is reached The median CPU usage (blue line) was dropping over time and the number of cores (bars) growing at a higher pace than the load Symptoms of a bottleneck
22. Our goals { }
23. Our goals { } Systems remain functional even if overloaded
24. Our goals Problematic systems can't affect others { } Systems remain functional even if overloaded
25. Seeing through the noise 03 { } .. ..
26. CPU isolation Linux config Nature Mechanism Prevents node saturation CPU requests CPU limits
27. CPU isolation Linux config Nature Mechanism Prevents node saturation CPU requests cpu.shares CPU limits cpu.quota
28. CPU isolation Linux config Nature Mechanism Prevents node saturation CPU requests cpu.shares Soft limit CPU limits cpu.quota Hard limit Hespanha
29. CPU isolation Linux config Nature Mechanism Prevents node saturation CPU requests cpu.shares Soft limit Prioritization when all CPUs are busy CPU limits cpu.quota Hard limit Enforced even if node has available CPU
30. CPU isolation Linux config Nature Mechanism Prevents node saturation CPU requests cpu.shares Soft limit Prioritization when all CPUs are busy No CPU limits cpu.quota Hard limit Enforced even if node has available CPU Yes** ** Partially since it's a quota and not a concurrency limit, but it's generally enough
31. CPU throttling oddness
32. CPU throttling oddness
33. CPU throttling intuition CPU quota: 4 cores Each square: 20ms of CPU
34. CPU throttling intuition CPU quota: 4 cores Each square: 20ms of CPU
35. CPU throttling intuition CPU quota: 4 cores Each square: 20ms of CPU
36. CPU throttling intuition CPU quota: 4 cores Each square: 20ms of CPU
37. CPU throttling intuition CPU quota: 4 cores Each square: 20ms of CPU
38. Current schools of thought { }
39. Current schools of thought CPU pinning is the one true way! Just disable limits! { }
40. • How can we prevent node saturation? • Environment-dependent performance Current schools of thought CPU pinning is the one true way! Just disable limits! { }
41. • How can we prevent node saturation? • Environment-dependent performance Current schools of thought • What about small systems with fractional quotas? • No possibility of bursts • k8s scheduling pressure CPU pinning is the one true way! Just disable limits! { }
42. • How can we prevent node saturation? • Environment-dependent performance Current schools of thought • What about small systems with fractional quotas? • No possibility of bursts • k8s scheduling pressure CPU pinning is the one true way! Just disable limits! { } What is it hiding? 🤔
43. The danger of averages
44. The danger of averages
45. The danger of averages
46. The danger of averages
47. The danger of averages Flavio
48. The danger of averages Flavio
49. The danger of averages
50. The danger of averages
51. The danger of averages What is it hiding? 🤔
52. The CPU avg is a lie! 😱
53. Fine-grained CPU metrics
54. Based on https://github.com/sqshq/sampler Fine-grained CPU metrics
55. Fine-grained CPU metrics Does not require a shorter Prometheus scrapping interval
56. How can we make systems behave within the CPU quota?
57. Building an adaptive threading model 04 { } .. ..
58. Nauvoo 01 02 03
59. You can't improve what you don't measure! Nauvoo 01 02 03 Fine-grained perf metrics
60. You can't improve what you don't measure! Nauvoo No more manual thread pool tuning, avoids CPU throttling on the fly 01 02 03 Fine-grained perf metrics Adaptive concurrency
61. You can't improve what you don't measure! Nauvoo No more manual thread pool tuning, avoids CPU throttling on the fly Rejects work above the system's capacity, avoids unbounded queuing and GC death spirals 01 02 03 Fine-grained perf metrics Adaptive concurrency Reactive backpressure
62. Detecting degradation
63. Detecting degradation v0: check all the things { } Hespanha
64. Detecting degradation Multiple checks: • CPU usage • Throttled % • Memory Tries to avoid degradation v0: check all the things { }
65. Detecting degradation Multiple checks: • CPU usage • Throttled % • Memory Tries to avoid degradation v1: heartbeat mode v0: check all the things { }
66. Detecting degradation • Inspired by jHiccup • while(true) { measure Thread.sleep(1) } • Allows a configurable level of degradation • Also detects GC pauses, safepoints, allocation stalls Multiple checks: • CPU usage • Throttled % • Memory Tries to avoid degradation v1: heartbeat mode v0: check all the things { }
67. Controlling degradation Linux Scheduler
68. Controlling degradation Linux Scheduler Executor Executor Executor
69. Controlling degradation Linux Scheduler Executor Executor Executor Nauvoo
70. Controlling degradation If there's degradation, reduce concurrency
71. Controlling degradation If there's degradation, reduce concurrency If threads are reliably scheduled, allow more concurrency Flavio
72. Main challenges - Reaction time Start with small changes and escalate via exponential steps - Control loop stability Introduce metastable state thresholds to stabilize changes Controlling degradation If there's degradation, reduce concurrency If threads are reliably scheduled, allow more concurrency
73. Main challenges - Reaction time Start with small changes and escalate via exponential steps - Control loop stability Introduce metastable state thresholds to stabilize changes Controlling degradation If there's degradation, reduce concurrency If threads are reliably scheduled, allow more concurrency
74. Main challenges - Reaction time Start with small changes and escalate via exponential steps - Control loop stability Introduce metastable state thresholds to stabilize changes Controlling degradation If there's degradation, reduce concurrency If threads are reliably scheduled, allow more concurrency
75. Demo Adapting # of threads to a load with low CPU usage and thread blocking Load increasing Number of Threads increasing ing to handle the load CPU Throttling under control
76. Demo Adapting # of threads to a CPU intensive load + rejections Number of Threads decreasing Service suffering with CPU Throttling Tasks being rejected
77. Results
78. Results Social - Stability improvement
79. Results Magnitude - Latency reduction
80. Results Stormshield - Cost reduction
81. The path ahead 05 { } .. ..
82. Further optimization
83. Further optimization Hespanha
84. Further optimization
85. Nauvoo v2 01 02 03
86. Nauvoo v2 01 02 03 Lower overhead
87. Nauvoo v2 01 02 03 Lower overhead Loom integration
88. Nauvoo v2 01 02 03 Lower overhead Loom integration Prepare for open source
89. CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik Optimizations by several teams! major rewrites, database migration, tunings, ... Flavio
90. CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik Optimizations by several teams! major rewrites, database migration, tunings, ... Nauvoo
91. Payday sanity
92. CREDITS: This presentation template was created by Slidesgo, and includes icons by Flaticon, and infographics & images by Freepik Thanks! @fbrasisil @luiz_hespanha