Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Autoscaling in Kubernetes

Avatar for Atsushi Tanaka Atsushi Tanaka
September 09, 2018

Autoscaling in Kubernetes

Avatar for Atsushi Tanaka

Atsushi Tanaka

September 09, 2018
Tweet

More Decks by Atsushi Tanaka

Other Decks in Programming

Transcript

  1. ©2018 Wantedly, Inc. • Atsushi Tanaka ాதಞࢤ  • Infrastructure

    Engineer
 at Wantedly, Inc. • Interested in k8s, Docker and Golang About @bgpat
  2. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS w $MVTUFS"VUPTDBMFS w 7FSUJDBM1PE"VUPTDBMFS w

    "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ ਫฏεέʔϧ ਨ௚εέʔϧ
  3. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  4. ©2018 Wantedly, Inc. /PEF /PEF /PEF Kubernetesͷεέʔϧ 1PE 4FSWJDF 1PE

    1PE 1PE 1PE %FQMPZNFOU
 3FQMJDB4FU Service ͸ Pod ʹରͯ͠
 ϦΫΤετΛϩʔυόϥϯε͢Δ Deployment ͸ Pod ͷ਺Λ
 Ұఆʹอͭ
  5. ©2018 Wantedly, Inc. εέʔϧ͢Δج४ w $16࢖༻཰ͷΈ BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͘ϝτϦΫε

    BVUPTDBMJOHW  w Ϧιʔεʹඥ෇͔ͳ͍ϝτϦΫε LTW Horizontal Pod Autoscaler
  6. ©2018 Wantedly, Inc. autoscaling/v2 ෳ਺ͷϝτϦΫεΛઃఆՄೳ ͲΕ͔ͻͱͭͰ΋ᮢ஋Λ௒͑Δͱεέʔϧ͢Δ • Resource: CPUɾϝϞϦ ͷ࢖༻཰

    • Object: Kubernetes ͕؅ཧ͢ΔϦιʔεͷϝτϦΫε • Pods: Pod ʹؔ͢ΔϝτϦΫε • External: Kubernetes ͷϦιʔεʹؔ࿈͠ͳ͍ϝτϦΫε Horizontal Pod Autoscaler
  7. ©2018 Wantedly, Inc. API Λ࢖ͬͯϝτϦΫεΛऔಘ͢Δ • Resource Metrics API •

    Custom Metrics API • External Metrics API Horizontal Pod Autoscaler
  8. ©2018 Wantedly, Inc. • Metrics Server or Heapster • Pod

    Λ Node ʹ഑ஔ͢Δͱ͖ʹ࢖ΘΕΔ • औΕΔϝτϦΫε͸ CPU ͱ ϝϞϦ ͷΈ Resource Metrics API
  9. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  10. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  11. ©2018 Wantedly, Inc. • Pod ͷਨ௚εέʔϧ • resource.limit ͱ resource.request

    Λมߋ • Node ΁ͷεέδϡʔϦϯά࣌ʹӨڹ • ᮢ஋͸ࣗಈઃఆ • HPAͱͷซ༻΋Մೳ Vertical Pod Autoscaler
  12. ©2018 Wantedly, Inc. w )PSJ[POUBM1PE"VUPTDBMFS )1"  w $MVTUFS"VUPTDBMFS w

    7FSUJDBM1PE"VUPTDBMFS w "EEPO3FTJ[FS KubernetesͰ࢖͑ΔΦʔτεέʔϧ
  13. ©2018 Wantedly, Inc. • kops Ͱ AWS ্ʹߏங • ϊʔυ਺:

    ໿40୆ • αʔϏε਺: ໿70ݸ • ΞϓϦέʔγϣϯͷ࣮૷ • Ruby on Rails • Goݴޠ • Python (ػցֶश) • C++, Rust (ը૾ॲཧ) WantedlyͷKubernetesΫϥελ
  14. ©2018 Wantedly, Inc. • HPA + Custom Metrics • Prometheus

    • k8s-prometheus-adapter • Prometheus Operator • nginx-exporter • Cluster Autoscaler WantedlyͰ࢖͍ͬͯΔΦʔτεέʔϧ
  15. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU manage manage metrics
  16. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  17. ©2018 Wantedly, Inc. Prometheus Operator apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata:

    name: service-monitor namespace: custom-metrics labels: service-monitor: prometheus spec: selector: matchLabels: custom-metrics: prometheus namespaceSelector: any: true endpoints: - port: http - port: https custom-metrics: prometheus ͷϥϕϧ͕෇͍͍ͯΔαʔϏε Λ؂ࢹ
  18. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  19. ©2018 Wantedly, Inc. Nginxͷstub_status location /basic_status { stub_status; } Active

    connections: 291 server accepts handled requests 16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 ଴ͪίωΫγϣϯ਺Λ࢖͏
  20. ©2018 Wantedly, Inc. Active connections: 291 server accepts handled requests

    16630948 16630948 31070465 Reading: 6 Writing: 179 Waiting: 106 nginx-exporter # HELP nginx_connections_current Number of connections currently processed by nginx # TYPE nginx_connections_current gauge nginx_connections_current{state="active"} 2 nginx_connections_current{state="reading"} 0 nginx_connections_current{state="waiting"} 1 nginx_connections_current{state="writing"} 1
  21. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  22. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU create create metrics
  23. ©2018 Wantedly, Inc. 1SPNFUIFVT 1SPNFUIFVT 1PE 1PE HPA + Custom

    Metrics ͷߏ੒ .BTUFS/PEF $VTUPN.FUSJDT "1*4FSWFS ,VCFSOFUFT "1*4FSWFS 1PE "QQMJDBUJPO /HJOY OHJOY FYQPSUFS proxy )1" metrics proxy stub_status 1SPNFUIFVT 1SPNFUIFVT 0QFSBUPS manage metrics %FQMPZNFOU scale 3FQMJDB4FU metrics manage manage
  24. ©2018 Wantedly, Inc. HPA apiVersion: autoscaling/v2beta1 kind: HorizontalPodAutoscaler metadata: name:

    wantedly spec: maxReplicas: 50 minReplicas: 20 scaleTargetRef: apiVersion: apps/v1 kind: Deployment name: wantedly metrics: - type: Resource resource: name: cpu targetAverageUtilization: 50 - type: Pods pods: metricName: nginx_connections_current{state="writing"} targetAverageValue: 2.5 CPU ͱ Nginx ͷ
 ଴ͪίωΫγϣϯͰεέʔϧ
  25. ©2018 Wantedly, Inc. • αʔόʔͷίετ͕཈͑ΒΕͨ • EC2 30୆ → Pod

    20୆ (EC2 7୆෼) • εύΠΫʹڧ͘ͳͬͨ(ؾ͕͢Δ) • Ξϥʔτ͕ݮͬͨ • ᮢ஋ܾΊ͕େม • ௨ৗ࣌ʹΪϦΪϦεέʔϧ͠ͳ͍஋Λ୳͢ KubernetesͷΦʔτεέʔϧΛಋೖͯ͠
  26. ©2018 Wantedly, Inc. • Custom Metrics Λಋೖ͢Δ • CPU͚ͩͷͱ͖ΑΓޮ཰Α͘Φʔτεέʔϧ͕࢖͑Δ •

    ΞϓϦέʔγϣϯͷىಈΛૣ͘͢Δ • ͢͹΍͘εέʔϧͯ͠΋ىಈ͕஗͍ͱεύΠΫʹ͸উͯͳ͍ • API Server ͷΦϓγϣϯΛมߋ͢Δ • --horizontal-pod-autoscaler-sync-period • ϞχλϦϯάͰ͖Δ࢓૊Έ • Wantedly Ͱ͸ DataDog Λ࢖͍ͬͯΔ KubernetesͷΦʔτεέʔϧಋೖͷTips