Автомасштабирование и управление ресурсами в Kubernetes

Доклад технического директора компании «Флант» (https://flant.ru/) Дмитрия Столярова на секции «DevOps» конференции «Стачка-2019» в Ульяновске.

* Текстовый обзор доклада: https://habr.com/company/flant/blog/459326/
* Видео с выступления: https://youtu.be/10ZR-fbyuSY

Transcript

1. Автомасштабирование и управление ресурсами в Kubernetes Дмитрий Столяров И Ленин такой молодой, И юный — Октябрь впереди!
2. Дмитрий Столяров CTO и соучредитель
3. Андрей Климентьев Андрей Сидоров
4. Андрей Климентьев Андрей Половов Андрей Сидоров
5. Антон Климов Андрей Климентьев Андрей Половов Андрей Сидоров
6. flant.ru habr.com/ru/company/flant youtube.com/c/Флант github.com/flant DevOps-профессионалы t.me/flant_ru
7. Автомасштабирование и управление ресурсами в Kubernetes
8. Автомасштабирование и управление ресурсами в Kubernetes
11. Повторяемость
12. Повторяемость Изоляция
16. ...
17. ...
18. ...
19. ...
20. ...
21. ...
22. ...
23. ...
24. ...
25. ...
26. ... Кластер контейнеров
27. ... Кластер контейнеров
28. ... Кластер контейнеров
29. ... 4 Кластер контейнеров Высокоуровневые примитивы
30. ... API 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API
31. ... API YAML 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API
32. Лучшие практики CI/CD с Docker youtu.be/FVFhqb1CbRc Наш опыт с Kubernetes в небольших проектах youtu.be/CgCLPYJRxbU Лучшие практики CI/CD с Kubernetes и GitLab youtu.be/G3nELxmECd8 Мониторинг и Kubernetes youtu.be/zj6SlzzBRaA Расширяем и дополняем Kubernetes youtu.be/6VHk1R1TNgk youtube.com/c/Флант Подписывайтесь на наш YouTube-канал!
33. ... API YAML 4 Кластер контейнеров Высокоуровневые примитивы Декларативное API
34. Автомасштабирование и управление ресурсами в Kubernetes
35. Автомасштабирование и управление ресурсами в Kubernetes
36. CPU
38. CPU
39. CPU $ htop ==============================
40. CPU $ htop ============================== nginx
41. CPU $ htop ============================== nginx php-fpm
42. CPU $ htop ============================== nginx php-fpm mysql
43. CPU $ htop ============================== nginx php-fpm mysql
44. CPU $ htop ============================== nginx: master nginx: worker nginx: worker php-fpm mysql
45. CPU $ htop ============================== nginx: master nginx: worker nginx: worker php-fpm mysql
46. CPU $ htop ============================== nginx: master nginx: worker nginx: worker php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql
47. CPU $ htop ============================== nginx: master nginx: worker nginx: worker php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
48. CPU $ htop ============================== nginx: master nginx: worker nginx: worker php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker mysql 5 71 74 6 205 217 213 209 460
49. CPU $ htop ============================== nginx: master nginx: worker nginx: worker 5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
50. А что такое Docker?
51. А что такое Docker? контейнер
52. А что такое контейнер?
53. capabilities 2.2 / 1999 А что такое контейнер?
54. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 А что такое контейнер?
55. capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 А что такое контейнер?
56. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013
57. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
58. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
59. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
60. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
61. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
62. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
63. overlay 2014 capabilities 2.2 / 1999 namespaces 2.6.19 / nov 2006 cgroups 2.6.24 / jan 2008 aufs 2007 kernel А что такое контейнер? docker 2013 rkt 2014 cri-o 2017 podman 2018 overlay2 2015 zfs 2013 btrfs 2013 veth 2007 ipvlan 2014 macvlan 2007
64. CPU $ htop ============================== nginx: master nginx: worker nginx: worker 5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
65. CPU $ htop ============================== nginx: master nginx: worker nginx: worker 5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460
66. CPU $ htop ============================== nginx: master nginx: worker nginx: worker 5 71 74 $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 $ htop ============================== mysql 460 1000
67. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 1000 150 850 460 5 71 74 6 205 217 213 209 460
68. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000
69. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 150 850 460 5 71 74 6 205 217 213 209 460 1000 333 333 333
70. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
71. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 333 333 333
72. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
73. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 333
74. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 + 333 91 +
75. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 183 333 150 - = 333 91 424 + = 333 91 424 + =
76. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5
77. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 850 460 2 3 5 300 200 500 1000
78. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 2 3 5 300 200 500
79. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 200 500 2 3 5 300
80. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 40 500 460 - = 2 3 5 300
81. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 500 460 - = 2 3 5
82. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 + 40 500 460 - = 2 3 5 50 +
83. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 1000 150 850 460 50 200 150 - = 300 40 390 + = 40 500 460 - = 2 3 5 50 +
84. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3
85. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 78 78 78 78 78
86. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
87. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 78 78 78 78 72 78 = -
88. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 72 78 = -
89. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== mysql 5 71 74 460 1000 150 460 50 200 150 - = 300 40 + = 40 500 460 - = 2 5 50 + $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker 6 205 217 213 209 850 390 3 6 96 96 96 96 2.12 2.26 2.21 2.17 100%
90. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 1000
91. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000
92. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 5 71 74 460
93. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 2 3 5 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100%
94. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
95. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 CFS = Completely Fair Scheduler
96. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5 Реквесты
99. nginx mysql php
100. nginx mysql php
101. nginx mysql php
102. 15 sec nginx mysql php
103. 15 sec nginx mysql php
104. 15 sec nginx mysql php
105. nginx mysql php
106. nginx mysql php
107. 39 sec nginx mysql php
108. 39 sec nginx mysql php
109. 39 sec nginx mysql php
110. 39 sec nginx mysql php
111. 39 sec nginx mysql php
112. 39 sec nginx mysql php
113. 39 sec nginx mysql php
114. 39 sec nginx mysql php
115. 39 sec nginx mysql php
116. 39 sec nginx mysql php
117. nginx mysql php
118. nginx mysql php
119. nginx mysql php 46 sec
120. nginx mysql php 46 sec
121. nginx mysql php 46 sec
122. nginx mysql php 46 sec
123. nginx mysql php 46 sec
124. nginx mysql php 46 sec
125. nginx mysql php 46 sec
126. nginx mysql php 46 sec
127. nginx mysql php
128. nginx mysql php
129. nginx mysql php
130. nginx mysql php Лимиты
131. nginx mysql php
132. nginx mysql php 46 sec
133. nginx mysql php 46 sec
134. nginx mysql php 46 sec
135. nginx mysql php 46 sec
136. nginx mysql php 46 sec
137. nginx mysql php 46 sec
138. nginx mysql php 46 sec Куплю машину ближе к светофору!
139. nginx mysql php 46 sec Куплю машину поближе к светофору!
140. CPU
141. kernel CPU
142. kernel CPU CFS
143. kernel CPU CFS cgroup
144. kernel CPU CFS cgroup
145. kernel CPU CFS cgroup
146. kernel CPU CFS cgroup quota
147. kernel CPU CFS cgroup quota shares
148. kernel CPU Крутилки CFS cgroup quota shares
149. kernel CPU Крутилки CFS cgroup quota shares Реквест Лимит
150. В чём мерить CPU?
151. в попугаях В чём мерить CPU?
152. в попугаях В чём мерить CPU? в процентах
153. в попугаях в процентах В чём мерить CPU? в весе
154. в попугаях в весе в процентах В чём мерить CPU? в секундах
155. в секундах в попугаях в весе в процентах В чём мерить CPU? в ядрах! N
156. ? N
157. ? N
158. ? N
159. ? N
160. ? N
161. ? N
162. ? N 500
163. ? N 500 1000
164. ? N 500 1000 1500
165. ? N 500 1000 1500 ? ? ?
166. ? N 500 1000 1500 ? ? ?
167. ? N 1000 1500 ? ? ? 500
168. ? N 1000 1500 ? ? ? 500 ( )
169. ? N ? ? ? 500 ( ) 1000 1500
170. ? N ? ? 1000 1500 ? 500 3000
171. ? N ? 1000 1500 ? ? 500 3000
172. ? N ? 1000 1500 ? ? 500 3000 3 x
173. ? N ? 1000 1500 ? 0.5 500 3000 3 x
174. ? N ? 1000 1500 ? 0.5 500
175. ? N ? 1000 1500 ? 0.5 500
176. ? N 1.5 1000 1500 1 0.5 500
177. ? N 1.5 1000 1500 1 0.5 500
178. ? N 1.5 1000 1500 1 0.5 500 ? 1500 ? 500 1000 ?
179. ? N 1000 1500 500 ? 1500 ? 500 1.5 1 0.5 1000 ?
180. ? N 1000 1500 500 3 1500 1 500 1.5 1 0.5 1000 2
181. ? N 1000 1500 500 1.5 1 0.5 3 1500 1 500 1000 2
182. ? N 1000 1500 500 1.5 1 0.5 3 1000 1500 2 1 500 3000 ?
183. ? N 1000 1500 500 1.5 1 0.5 3 1000 1500 2 1 500 3000 3
184. ? N 1000 1500 500 1.5 1 0.5 3 1000 1500 2 1 500 3000 3
185. ? N 1000 1500 500 1.5 1 0.5 1.5 1000 1500 1 0.5 500 3000 3
186. ? N 1000 1500 500 1.5 1000 1500 1 0.5 500 1.5 1 0.5 3000 3
187. ? N 1 миллиядро = 1 cpu share* *0 миллиядер = 2 cpu share *На самом деле 1000 миллиядер = 1024 cpu shares
190. allocatable: cpu: "8"
191. allocatable: cpu: "8"
192. allocatable: cpu: "8"
193. allocatable: cpu: "8"
194. allocatable: cpu: "8"
195. allocatable: cpu: "4" allocatable: cpu: "8"
196. allocatable: cpu: "4" allocatable: cpu: "8"
197. allocatable: cpu: "4" allocatable: cpu: "6" allocatable: cpu: "8"
198. allocatable: cpu: "4" allocatable: cpu: "6" allocatable: cpu: "8"
201. cpu: 2500m
202. cpu: 2500m
203. cpu: 2500m
204. cpu: 2500m
205. А что если мы не указали никакой request?
206. 100% ресурсов
207. Request = Limit Request = Limit Guaranteed
208. Request = 100m Limit = 200m Request = Limit Request = Limit Burstable Guaranteed
209. Request = 100m Limit = 200m Request = Limit Request = Limit Burstable Guaranteed
210. Request = 100m Limit = 200m Limit = 200m Request = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Guaranteed
211. Request = 100m Limit = 200m Limit = 200m Request = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
213. Memory
214. kernel CPU Крутилки CFS cgroup quota shares Реквест Лимит
215. kernel CPU CFS cgroup quota shares Реквест Лимит
216. kernel CPU CFS cgroup quota shares kernel Memory mm subsystem cgroup limit_in_bytes oom_score_adj Реквест Лимит Реквест Лимит
217. kernel CPU cgroup quota shares kernel Memory cgroup limit_in_bytes oom_score_adj Реквест Лимит Реквест Лимит CFS mm subsystem
218. kernel Memory Крутилки mm subsystem cgroup limit_in_bytes oom_score_adj Реквест Лимит
219. kernel Memory Крутилки mm subsystem cgroup limit_in_bytes oom_score_adj Реквест Лимит
237. А как же вычисляется oom_score_adj?
238. Request = 100m Limit = 200m Limit = 200m Request = 100m Request = 100m Limit = 200m Limit = 200m Request = 100m Request = Limit Request = Limit Burstable Best Effort Guaranteed
239. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class
240. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class oom_score_adj
241. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class oom_score_adj -998
242. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class oom_score_adj -998 1000
243. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000
244. Burstable Best Effort Guaranteed Request = Limit Request = 100m Limit = 200m QoS Class oom_score_adj -998 min(max(2, 1000 - (1000*memoryRequestBytes)/ machineMemoryCapacityBytes), 999) 1000 Обратно пропорционально request ÷ capacity
245. Реквест Лимит kernel Memory mm subsystem cgroup limit_in_bytes oom_score_adj Крутилки
246. Реквест Лимит kernel Memory mm subsystem cgroup limit_in_bytes oom_score_adj Крутилки
259. Итого...
260. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
261. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
262. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
263. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
264. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
265. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
266. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
267. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
268. requests: cpu: 250m memory: 5G limits: cpu: 500m memory: 10G
269. Автомасштабирование и управление ресурсами в Kubernetes
270. Автомасштабирование и управление ресурсами в Kubernetes
271. Автомасштабирование и управление ресурсами в Kubernetes
272. Автомасштабирование и управление ресурсами в Kubernetes
273. Автомасштабирование и управление ресурсами в Kubernetes
274. Автомасштабирование и управление ресурсами в Kubernetes
278. cpu: 2500m
279. cpu: 2500m Pending...
280. cpu: 2500m Pending...
281. allocatable: cpu: "8" cpu: 2500m Pending...
282. cpu: 2500m Pending...
283. cpu: 2500m Pending...
286. (Авто)масштабирование кластера
300. Но...
302. Нельзя так просто взять и...
306. Deployment 6
307. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms 6 app-d38nl app-vq1g5
308. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms 6 app-d38nl app-vq1g5
309. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6 app-d38nl app-vq1g5
310. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6 app-d38nl app-vq1g5
311. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green 6 app-d38nl app-vq1g5
312. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-d38nl app-vq1g5 6
313. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-d38nl app-vq1g5 6
314. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-heb0y app-my9sq 6
315. app-59ptg app-xfvr7 app-ck9cm Deployment app-4ktms $ kubectl drain node-green app-heb0y app-my9sq 6
318. Statefulset 3
319. Statefulset 3 mongo-1 mongo-3 mongo-2
320. Statefulset 3 mongo-1 mongo-3 mongo-2
321. Statefulset 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff
322. Statefulset 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff
323. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff
324. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff
325. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff
326. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 mongo-2 CrashLoopBackOff PodDisruptionBudget minAvailable: 2
327. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2 CrashLoopBackOff
328. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2 CrashLoopBackOff
329. Statefulset $ kubectl drain node-green 3 CrashLoopBackOff mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2
330. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2 CrashLoopBackOff
331. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2 CrashLoopBackOff
332. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2
333. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2
334. Statefulset $ kubectl drain node-green 3 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2 mongo-2
335. Statefulset $ kubectl drain node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
336. Statefulset $ kubectl drain node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
337. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
338. Statefulset $ kubectl drain node-green $ kubectl uncordon node-green 3 mongo-2 mongo-1 mongo-3 PodDisruptionBudget minAvailable: 2
340. 1. Пересоздание
341. 1. Пересоздание 2. PodDisruptionBudget
342. ... 3-10 backend
343. ... 3-10 backend stateful
344. ... 3-10 backend stateful frontend
345. ... dev ... 3-10 backend stateful frontend 5-20
346. ... dev ... 0-30 test ... 3-10 backend stateful frontend 5-20
347. ... dev ... 0-30 test ... 3-10 backend stateful frontend 5-20
348. (Авто)масштабирование кластера
360. 3 Ручное масштабирование?
361. 3 Ручное масштабирование?
383. 3 Горизонтальное (авто) масштабирование
384. Что именно измерять?
385. Что именно измерять? Как интерпретировать полученные значения?
386. CPU Usage CPU Throttling Memory Usage OOM Системные RPS Connections Worker Pool Usage Уровня приложения Queue Length Response Time
387. USE Utilization – потребление Saturation – пресыщение Errors – ошибки
388. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование
392. requests: cpu: 500m 3
393. 3 requests: cpu: 500m
394. 3 requests: cpu: 500m
395. 3 requests: cpu: 500m
396. 3 requests: cpu: 1000m
397. requests: cpu: 1000m 3
398. requests: cpu: 1000m 3
399. requests: cpu: 1000m 3
400. requests: cpu: 1000m 3
401. requests: cpu: 1000m 3
402. requests: cpu: 1000m 3
403. requests: cpu: 1000m 3
404. requests: cpu: 1000m 3
405. 6 3 requests: cpu: 1000m Вертикальное (авто) масштабирование
406. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование
407. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
408. CPU $ htop ============================== nginx: master nginx: worker nginx: worker $ htop ============================== php-fpm: master php-fpm: worker php-fpm: worker php-fpm: worker php-fpm: worker $ htop ============================== mysql 5 71 74 6 205 217 213 209 460 150 390 460 1000 6 96 96 96 96 2.12 2.26 2.21 2.17 100% 5 71 74 460 100% 100% 100% 100% 2 3 5
409. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование
410. 6 3
411. 6 3
412. 6 3
413. 6 3 requests: cpu: 1000m
414. 6 3 requests: cpu: 1000m
415. 6 3 requests: cpu: 1000m
416. 6 3 requests: cpu: 1000m
417. 6 3 requests: cpu: 500m
418. 6 3 requests: cpu: 500m
419. 6 3 requests: cpu: 500m
420. 6 3 requests: cpu: 500m
421. 6 3 requests: cpu: 500m
422. 6 3 requests: cpu: 500m
423. 6 3 requests: cpu: 500m
424. 6 5 requests: cpu: 500m
425. 6 5 requests: cpu: 500m
426. 6 5 requests: cpu: 500m
427. (Авто)масштабирование кластера Горизонтальное (авто)масштабирование Вертикальное (авто)масштабирование
429. max(utilisation) при min(saturation)
430. Спасибо! Наш блог на Хабре habr.com/company/flant Наш YouTube-канал youtube.com/c/Флант shell-operator github.com/flant/shell-operator addon-operator github.com/flant/addon-operator werf github.com/flant/werf grafana-statusmap github.com/flant/grafana-statusmap kubedog github.com/flant/kubedog Дмитрий Столяров dmitry.stolyarov@flant.com CTO и соучредитель