Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Автомасштабирование и управление ресурсами в Ku...

flant
April 27, 2019

Автомасштабирование и управление ресурсами в Kubernetes

Доклад технического директора компании «Флант» (https://flant.ru/) Дмитрия Столярова на секции «DevOps» конференции «Стачка-2019» в Ульяновске.

* Текстовый обзор доклада: https://habr.com/company/flant/blog/459326/
* Видео с выступления: https://youtu.be/10ZR-fbyuSY

flant

April 27, 2019
Tweet

More Decks by flant

Other Decks in Technology

Transcript

  1. ...

  2. ...

  3. ...

  4. ...

  5. ...

  6. ...

  7. ...

  8. ...

  9. ...

  10. ...

  11. Лучшие практики CI/CD с Docker youtu.be/FVFhqb1CbRc Наш опыт с Kubernetes

    в небольших проектах youtu.be/CgCLPYJRxbU Лучшие практики CI/CD с Kubernetes и GitLab youtu.be/G3nELxmECd8 Мониторинг и Kubernetes youtu.be/zj6SlzzBRaA Расширяем и дополняем Kubernetes youtu.be/6VHk1R1TNgk youtube.com/c/Флант Подписывайтесь на наш YouTube-канал!
  12. CPU

  13. CPU

  14. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql
  15. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
  16. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
  17. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  18. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 А

    что такое контейнер?
  19. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups

    2.6.24 / jan 2008 А что такое контейнер?
  20. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013
  21. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  22. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  23. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  24. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  25. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  26. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  27. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov

    2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
  28. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  29. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
  30. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460 1000
  31. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 1000 150 850 460 5 71 74 6 205 217 213 209 460
  32. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000
  33. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000 333 333 333
  34. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
  35. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
  36. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
  37. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
  38. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 + 333 91 +
  39. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 424 + = 333 91 424 + =
  40. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5
  41. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 850 460 2 3 5 300 200 500 1000
  42. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5 300 200 500
  43. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 200 500 2 3 5 300
  44. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 40 500 460 - = 2 3 5 300
  45. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 500 460 - = 2 3 5
  46. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 + 40 500 460 - = 2 3 5 50 +
  47. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 390 + = 40 500 460 - = 2 3 5 50 +
  48. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3
  49. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 78 78 78 78 78
  50. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
  51. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
  52. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 72 78 = -
  53. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 2.12 2.26 2.21 2.17 100%
  54. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 1000
  55. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000
  56. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 5 71 74 460
  57. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100%
  58. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  59. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 CFS = Completely Fair Scheduler
  60. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 Реквесты
  61. CPU

  62. ? N

  63. ? N

  64. ? N

  65. ? N

  66. ? N

  67. ? N

  68. ? N 1000 1500 500 1.5 1 0.5 1.5 1000

    1500 1 0.5 500 3000 3
  69. ? N 1 миллиядро = 1 cpu share* *0 миллиядер

    = 2 cpu share *На самом деле 1000 миллиядер = 1024 cpu shares
  70. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Guaranteed
  71. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
  72. kernel CPU CFS cgroup quota shares kernel Memory mm subsystem

    cgroup limit_in_bytes oom_score_adj Реквест Лимит Реквест Лимит
  73. kernel CPU cgroup quota shares kernel Memory cgroup limit_in_bytes oom_score_adj

    Реквест Лимит Реквест Лимит CFS mm subsystem
  74. Request = 100m Limit = 200m Limit = 200m Request

    = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
  75. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998
  76. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 1000
  77. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000
  78. Burstable Best Effort Guaranteed Request = Limit Request = 100m

    Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000 Обратно пропорционально request ÷ capacity
  79. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3

    mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
  80. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3

    mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
  81. 3

  82. 3

  83. 3

  84. 3

  85. 3

  86. 3

  87. 3

  88. 3

  89. 3

  90. 3

  91. 3

  92. 3

  93. 3

  94. 3

  95. 6

  96. 6

  97. 6

  98. 6

  99. 6

  100. 6

  101. 6

  102. 6

  103. 6

  104. 6

  105. 6

  106. 3

  107. 3

  108. 3

  109. 3

  110. 3

  111. CPU Usage CPU Throttling Memory Usage OOM Системные RPS Connections

    Worker Pool Usage Уровня приложения Queue Length Response Time
  112. 3

  113. 3

  114. 3

  115. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  116. CPU $ htop ============================== nginx: master nginx: worker nginx: worker

    $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
  117. 6 3

  118. 6 3

  119. 6 3

  120. Спасибо! Наш блог на Хабре habr.com/company/flant Наш YouTube-канал youtube.com/c/Флант shell-operator

    github.com/flant/shell-operator addon-operator github.com/flant/addon-operator werf github.com/flant/werf grafana-statusmap github.com/flant/grafana-statusmap kubedog github.com/flant/kubedog Дмитрий Столяров dmitry.stolyarov@flant.com CTO и соучредитель