Scaling In Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes

Transcript

1. #KubeCon #CloudNativeCon Scaling In Kubernetes Safely on On-Prem KaaS Across 1,300+ Clusters and 40,000+ Nodes Private / On-Prem / OpenStack Shota Yoshimura Senior Platform Engineer, LY Corporation
2. Who Am I? • Senior Platform Engineer, LY Corporation Building & operating our Kubernetes as a Service platform • In my free time: anime & manga — into Demon Slayer (鬼滅の刃), Jujutsu Kaisen (呪術廻戦), and Frieren (葬送のフリーレン) • • Shota Yoshimura | @yosshi_ Also watched: the FIFA World Cup 2026 Favorite release logo: Kubernetes v1.36 "Haru" (ハル)
3. What KaaS Is
4. Where We Started — 2016: VMs on OpenStack Our Challenges • Provisioning environments often took more than a week • Nodes stayed unpatched for long periods — a security risk • Node failures meant on-call pages — often at 3 a.m. So we set out to build declarative infrastructure.
5. Kubernetes as a Service (KaaS) Architecture Kubernetes as a Service API Req Custom Resource apiVersion: zlab.co.jp/v1 kind: KubernetesCluster metadata: - name: foo # Control Plane controlPlaneMachineGroups: - name: cp flavor: 4v-8G-64G replicas: 2 # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G replicas: 3 Apply KubernetesCluster Watch name: foo KubernetesCluster Controller Create Create MachineDeployment Watch name: foo-<group> Create MachineDeployment Controller Control Plane Nodes VM MachineSet Watch name: foo-<group>-<hash> # Ingress - name: ingress flavor: 4v-8G-64G replicas: 2 MachineSet Controller VM Worker Nodes Create VM Machine name:foo-<group>-<hash>-<rand> Watch Machine Controller VM VM Ingress Nodes VM Inspired by Kubernetes' own Deployment — applied to VMs. VM
6. How Traffic Reaches a Pod apiVersion: zlab.co.jp/v1 kind: KubernetesCluster metadata: - name: foo Apply # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G replicas: 3 # Ingress - name: ingress flavor: 4v-8G-64G replicas: 2 Kubernetes as a Service API Req API Req Software Load Balancer Create Create Create Client VIP Ingress Controller Pod Ingress Node Worker Node
7. Scaling Up and Out Are Easy Node Node PATCH { "flavor": "8v-32G128G" } Node Scale Up Node Node # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G replicas: 2 KaaS Scale Out PATCH { "replicas": "3" } Node Node Node
8. Node Failures Are Routine, Not Incidents 5–30 nodes fail somewhere in our fleet — every single day. (Well under 0.1% of the fleet) # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G No page. No manual recovery. replicas: 2 New Create KaaS Node Probe Node Node Failed Node Phase 1 Node fails KaaS New Failed Node KaaS Node Delete Node Phase 2 New node added Phase 3 Failed node removed
9. Batteries Included A vanilla Kubernetes cluster ships with none of this. Every cluster ships with all of this, already installed: Networking Observability nghttpx Ingress Controller kube-state-metrics Prometheus logging-agent CoreDNS metrics-server Alertmanager eventrouter node-local-dns Node Exporter Grafana ephemeral-storage-exporter Storage & GPU trident-operator NVIDIA k8s-device-plugin dcgm-exporter ...and more
10. Self-Service, Documented Every operational scenario has a runbook or guide. Teams are expected to handle it themselves. Platform team's job: keep the guardrails, not the pager.
11. Ten Years Later — 2026 Scale grew without linear team growth. 1,300+ clusters 40,000+ nodes ~1M containers 700+ clusters Operated by 15 engineers 400+ clusters 5 clusters 2017 20 clusters 2018 2019 2020 …. 2026 Today, one platform engineer operates 90 clusters.
12. Ten Years Later — 2026 Scale grew without linear team growth. 1,300+ clusters 40,000+ nodes ~1M containers 700+ clusters Operated by 15 engineers 400+ clusters 5 clusters 2017 20 clusters 2018 2019Scan for 2020 Case Study …. 2026 Today, one platform engineer operates 90 clusters.
13. Why Scale-In, and How
14. Hardware Isn't Getting Cheaper DDR5 is now ~5x pricier than just a year ago. Data: Stanford Memory Prices Project — dam.stanford.edu/memory-prices.html Our next hardware purchase became significantly more expensive.
15. And the Yen Isn't Helping The yen has weakened ~13% against the dollar in the same year. Data: FRED (Federal Reserve Economic Data), series DEXJPUS Buying more isn't the whole plan anymore. Using what we already have is too.
16. The Overprovisioning Problem A problem we created ourselves. • We kept usage fees low to make the platform easy to adopt. With little incentive to release capacity, overprovisioning was a rational choice. • Result: clusters that were created, then quietly stopped being used — especially in Dev environments. Raising fees won't fix this in time — and on-prem, charging more for hardware we already own doesn't change our real cash outflow.
17. Two Goals of Reclaiming Capacity Not just cost-cutting. • Keep headroom available — so teams can burst-scale fast during traffic spikes (campaigns, events). • Understand real surplus — so future hardware purchases are sized correctly, and capex isn't wasted.
18. LowUsageNodeReducer Here's what we built. It checks usage against Prometheus, and quietly reduces replicas on clusters that don't need what they have. Kubernetes as a Service Prometheus Check usage LowUsageNodeReducer PATCH { "replicas": "current - 1" } # Worker workerMachineGroups: - name: worker flavor: 4v-16G-64G replicas: 3 !> 2
19. Why Not Use Cluster Autoscaler? Cluster Autoscaler solves a different problem: responding to changing Pod demand within each cluster. Our goal was fleet-wide capacity reclamation: • Centrally managed across 1,300+ clusters • No per-team configuration • Conservative, long-term usage analysis • Better physical capacity and procurement planning
20. Deciding the Conditions: Conservative by Design A custom controller checks, before acting: • Node usage (CPU/memory/disk) under 30%, over a full 7-day peak. Most services follow a weekly cycle — a full week catches that pattern. • Machine-group request-to-allocatable ratio (CPU/memory/storage) under 50%. Even with low usage, high Pod requests can leave no room to schedule — new Pods would go Pending. • Minimum replica guarantee — 6 nodes by default. We run 3 AZs, 2 nodes per AZ — so a node failure doesn't leave AZ-aware Pods Pending. • A 3-day cooldown after every reduction. After a reduction, the cluster's state can keep shifting — the cooldown gives it time to settle before we look again.
21. Transparent, Not Silent Teams don't need to act — but they can always see what happened: Dashboard KubernetesCluster Annotations apiVersion: zlab.co.jp/v1 kind: KubernetesCluster metadata: - name: foo annotations: low-usage-node-reducer/last-reduced-at: "2026-07-10T08:02:00Z" low-usage-node-reducer/last-reduced-from-to: "14!>13" A Kubernetes Event [LowUsageNodeReducer] reduced worker machineGroup 'default' (flavor 4v-1G-64G) replicas from 14 to 13 on cluster 'foo'
22. Making Scale-In Safe
23. Without Guardrails, This Happens Removing a node safely takes more than just deleting it. Please... not my node! Phippy & friends — CNCF, CC-BY — phippy.io
24. Graceful Shutdown Comes First Pods Have to Be Able to Shut Down Gracefully After GracePeriodSeconds, if still not terminated (default 30s) Pod termination begins preStop hook SIGTERM preStop processing New connections SIGKILL SIGTERM processing No new connections from this point on Established connections The EndpointSlice controller marks the Pod not-ready kube-proxy propagates that to iptables, and new connections stop Guidance for every team • • preStop sleep — required. Graceful shutdown on SIGTERM — required.
25. Controlling Pod Exit Order on a Node Sequential taints control the shutdown order. Application Pods exit first, while logging and network components stay available to support a safe shutdown. Taint KaaS Taint Taint Taint:NoExecute Taint:NoExecute Taint:NoExecute Taint:NoExecute Taint:NoExecute Evicted App Taint:NoExecute Evicted Log Log Forwarding Forwarding Evicted Network Network Network Phase 1 App Phase 2 Log Forwarding Phase 3 Network e.g. Fluent Bit e.g. node-local-dns
26. PDBs Have to Be Set Correctly A PodDisruptionBudget can silently block node removal entirely. A common trap: percentages round up. 3 replicas, minAvailable: 70% → 3 × 0.7 = 2.1 → rounds up to 3 → disruptionsAllowed stuck at 0. Guardrail — we alert on exactly that: - alert: PDBDisruptionsAllowedZero expr: | (kube_poddisruptionbudget_status_pod_disruptions_allowed != 0) and on (namespace, poddisruptionbudget) # skip PDBs that are unhealthy only due to an in-progress rollout (kube_poddisruptionbudget_status_current_healthy != kube_poddisruptionbudget_status_desired_healthy) and on (namespace, poddisruptionbudget) (kube_poddisruptionbudget_status_desired_healthy > 0)
27. Pod Requests Have to Be Set Without accurate requests, the scheduler can't tell how much room a node really has. e.g. overcommitted memory → the OOM Killer. e.g. overcommitted disk → DiskPressure evictions. Guardrail — an alert finds Pods with no requests set. - alert: PodMemoryRequestNotSet expr: | group by (namespace, pod) (kube_pod_info) unless group by (namespace, pod) (kube_pod_resource_request{resource="memory"}) This is the value the scheduler itself considers — init containers and sidecars included.
28. Pod Requests Have to Be Tuned, Too Resource usage tends to grow the longer an application runs in production. Requests set once at launch drift out of date. Guardrail — an alert fires when usage diverges from requests. - alert: ExceedResourceRequestMemory expr: | sum by (namespace, pod) ( container_memory_working_set_bytes{container!~"POD|", image!=""} ) / sum by (namespace, pod) ( kube_pod_resource_request{resource="memory"} ) > 1
29. Closing the Ephemeral Storage Gap Per-Pod ephemeral-storage usage is not exposed in the same way as CPU and memory. So we built ephemeral-storage-exporter. It collects per-Pod usage from the kubelet Summary API on every node. ephemeral_storage_exporter_usage_bytes{namespace="prod",node="node-1",pod_name="app-abc123",pod_uid="uid-1"} 1.261e+10 Recent trend: Jobs and Image Volumes have made large unpacked images increasingly common. e.g. a real image we saw: 6.97 GB packed → 12.61 GB unpacked (~1.8x). Guidance, for images over 1 GiB: requests.ephemeral-storage = expected disk usage + unpacked image size.
30. Two Kinds of Guardrails Guardrails matter for running a platform. Here's how we typically split them. • CEL-based ValidatingAdmissionPolicy (VAP) → checks a single resource in isolation e.g. manifest misconfigurations, security-related settings • AlertRule → checks cluster-wide, dynamic state e.g. PodDisruptionBudget's disruptionsAllowed, Pod resource requests
31. Why This Is Safe at Scale Every cluster already does rolling node replacement, aligned to Kubernetes minor-version EOL. v1.34 EOL:2026-10-27 v1.35 EOL:2027-02-28 v1.36 EOL:2027-06-28 Ten years ago, every rolling update brought a wave of "nothing changed, but it broke" tickets. Today, treating nodes as cattle, not pets, is just part of our culture. Scale-in isn't a new risk. It's the same rolling operation we already trust.
32. Early Operational Results Development Environments • ~20% Projected Capacity Reduction • 0 Incidents • 0 User Inquiries (Baseline: 300–400 support tickets per year) Production Environments • Evaluating a cautious rollout
33. Summary
34. Three Things to Remember • On-prem, scale-in is about procurement and consolidation, not an hourly cloud bill. • The mechanism is trivial (fewer replicas). The safety model (usage, PDBs, requests, guardrails) is where the real engineering is. • Trust comes from what you already do. We didn't invent new risk. We reused rolling replacement.
35. Thank You Scan for Q&A Phippy & friends — CNCF, CC-BY — phippy.io