コンテナのメトリクスと モニタリングパターン / 190320-sakura-event

1. ίϯςφͷϝτϦΫεͱ ϞχλϦϯάύλʔϯ 2019/03/20 ͸ͯͳ×͘͞Β͕ߟ͑ΔςΫϊϩδʔͷະདྷ ʙίϯςφɾ෼ࢄܕσʔληϯλʔʙ #͘͞ΒͷΠϕϯτ
2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳
   
   Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
   IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ 
3. ؂ࢹͱϝτϦΫε
4. ؂ࢹͷ໨త • γεςϜ͕݈શʹಈ͍͍ͯΔ͔ΛνΣοΫ͢Δ • ظ଴ͨ͠ͱ͓Γʹಈ͍͍ͯΔ͔ • Ձ஋ΛੜΈग़͍ͯ͠Δ͔ • Ϧιʔεʹաෆ଍͸ͳ͍͔ • ো֐ͷ༧ஹ͸ͳ͍͔ • →ϝτϦΫεͷऔಘ/ऩूɺ؂ࢹɺΞϥʔτͰ࣮ݱ͢Δ
5. 2ͭͷλΠϓͷϝτϦΫε • 2ͭͷλΠϓͷϝτϦΫεʹ෼͚ͯߟ͑Δ • ΞϓϦέʔγϣϯϝτϦΫε • ϦιʔεϝτϦΫε
6. ΞϓϦέʔγϣϯϝτϦΫε • γεςϜ͕ਖ਼ৗʹಈ͍ͯΔ౓߹͍Λࣔ͢ • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ • γεςϜͷਖ਼ৗੑ΍Ձ஋ͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͸ো֐ݪҼڀ໌ʹ͸໾ཱͪʹ͍͘
7. ϦιʔεϝτϦΫε • γεςϜ͕ґଘ͢ΔϦιʔεͷ࢖༻ঢ়گΛࣔ͢ • Ϧιʔε࢖༻ྔ/཰ɺ๞࿨཰ɺΤϥʔ਺ͳͲ • ো֐ݪҼڀ໌΍ΩϟύγςΟϓϥϯχϯάͷࢦඪͱͳΔ • ͜ͷϝτϦΫε͚ͩͰ͸γεςϜ͕ਖ਼ৗʹಈ͍͍ͯΔ͔෼͔Βͳ͍
8. ͲͪΒ΋ॏཁ • ΞϓϦέʔγϣϯϝτϦΫεˠϦιʔεϝτϦΫεͷॱʹௐࠪ͢Δ • ଞͷΞϓϦέʔγϣϯʹґଘ͍ͯ͠Δ৔߹͸ಉ͡Α͏ʹௐࠪ͢Δ • ΢ΣϒαʔόɺΞϓϦέʔγϣϯαʔόɺσʔλϕʔεαʔό
9. ίϯςφͷϝτϦΫε
10. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε • ΞϓϦέʔγϣϯʹΑͬͯऔಘͰ͖ΔϝτϦΫε͸͞·͟· • ϦΫΤετ਺ɺ੒ޭ/ࣦഊɺϨεϙϯελΠϜɺεϧʔϓοτͳͲ
11. ίϯςφͷΞϓϦέʔγϣϯϝτϦΫε औಘํ๏ • ϑΝΠϧ • ϩά • ΤϯυϙΠϯτ
12. ϑΝΠϧ • σʔλϘϦϡʔϜ • ίϯςφؒͰϑΝΠϧ΍σΟϨΫτϦΛڞ༗ • Bind mounts • ίϯςφϗετͱϑΝΠϧ΍σΟϨΫτϦΛڞ༗
13. $ docker run -v $(pwd):/data alpine /bin/sh -c 'echo "hello world" >/data/hello' $ cat hello hello world e.g. Bind mounts
14. ϩά • ΞϓϦέʔγϣϯͷSTDOUT • docker logs ίϚϯυ • kubectl logs ίϚϯυ • ίϯςφ಺ͷϩάϑΝΠϧ • σʔλϘϦϡʔϜ, Bind mountsΛར༻
15. $ CONTAINER=$(docker run -d alpine /bin/sh -c 'while true; do date; sleep 1; done') $ docker logs --since 3s $CONTAINER Tue Mar 19 10:07:57 UTC 2019 Tue Mar 19 10:07:58 UTC 2019 Tue Mar 19 10:07:59 UTC 2019 e.g. docker logs
16. ΤϯυϙΠϯτ • ωοτϫʔΫܦ༝ͰΞΫηεՄೳͳΤϯυϙΠϯτ • ΞϓϦέʔγϣϯͰ࣮૷͢Δඞཁ͕͋Δ • /metrics, /stats, /health
17. $ curl http://${CONTAINER_IPADDR}/nginx_status Active connections: 43 server accepts handled requests 7368 7368 10993 Reading: 0 Writing: 5 Waiting: 38 e.g. Nginxͷstub_status
18. ίϯςφͷϦιʔεϝτϦΫε • CPU • Memory • I/O • ωοτϫʔΫ
19. ίϯςφͷϦιʔεϝτϦΫεऔಘํ๏ • Ծ૝ϑΝΠϧ • API • CLI
20. Ծ૝ϑΝΠϧ • cgroupͷstatϑΝΠϧ • cpu/cpuacct, memory, blkioαϒγεςϜͳͲ • /proc/PID/net/dev
21. $ cat /sys/fs/cgroup/memory/docker/${CONTAINER}/memory.stat cache 8192 rss 192512 rss_huge 0 mapped_file 0 dirty 0 [...] e.g. memory.stat
22. API • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελͷAPIͳͲ • Docker stats API (Docker) • Kubernetes/kubelet API (Kubernetes) • Task Metadata Endpoint (ECS) • cAdvisor API
23. $ curl -s --unix-socket \ /var/run/docker.sock http://localhost/${CONTAINER}/stats?stream=false | \ jq . [...] "cpu_stats": { "cpu_usage": { "total_usage": 963481062, "percpu_usage": [ 431767249, 531713813 ], [...] e.g. Docker stats API
24. CLI • ίϯςφϥϯλΠϜ΍ίϯςφΫϥελ͕ఏڙ͢ΔίϚϯυ • docker stats ίϚϯυ • kubectl top ίϚϯυ
25. $ docker stats --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d7f9c5c0acfc app 0.00% 1.945MiB / 1.952GiB 0.10% 1.5kB / 0B 5.11MB / 0B 2 8c484a94aca7 monitor 0.19% 652KiB / 1.952GiB 0.03% 1.41kB / 0B 1.24MB / 0B 2 e.g. docker stats
26. ωοτϫʔΫϝτϦΫεͷ஫ҙ఺ • /proc/PID/net/dev • ΠϯλʔϑΣʔε୯ҐͷϝτϦΫε
27. $ cat /proc/${CONTAINER_PID}/net/dev Inter-| Receive | Transmit face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed lo: 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 ecs-eth0: 467809174 273650 0 0 0 0 0 0 27453834 328421 0 0 0 0 0 0 eth0: 30877420 169783 0 0 0 0 0 0 55160434 183407 0 0 0 0 0 0 e.g. /proc/PID/net/dev
28. e.g. DockerͷωοτϫʔΫϞʔυ
29. e.g. DockerͷωοτϫʔΫϞʔυ • bridge, none • Private Network Mode • ίϯςφ͝ͱʹಠཱͨ͠ΠϯλʔϑΣʔε • host, container • Shared Network Mode • ϗετϓϩηε΍ଞͷίϯςφͱڞ௨ͷΠϯλʔϑΣʔε
30. ίϯςφΫϥελʹ͓͚Δ ϝτϦΫεͷऩू
31. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ
32. ίϯςφΫϥελͰϝτϦΫεΛͲ͏ूΊΔ͔ʁ • ίϯςφ͸εέδϡʔϥʹΑΓద੾ͳϗετʹσϓϩΠ͞ΕΔ • ϗετ͸εέʔϧ͢Δ • ίϯςφ಺ͰϞχλϦϯάΤʔδΣϯτΛಉډͤ͞Δͷ͸NG
33. ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάͷύλʔϯ • per-Host Monitoring Agent • Sidecar Monitoring Agent
34. per-Host Monitoring Agent
35. per-Host Monitoring Agent • ϗετ୯ҐͰίϯςφΛ؂ࢹ͢Δύλʔϯ • ֤ϗετʹϞχλϦϯάΤʔδΣϯτίϯςφΛσϓϩΠ͢Δ • ϗετͰಈ͘ίϯςφҰཡ͔Βಈతʹ؂ࢹઃఆΛੜ੒͢ΔͨΊɺ ෳࡶͳ࣮૷͕ඞཁ
36. Sidecar Monitoring Agent
37. Sidecar Monitoring Agent • ίϯςφ୯ҐͰ؂ࢹ͢Δύλʔϯ • ؂ࢹର৅ͷίϯςφʹԣ෇͚(Sidecar)͢ΔΑ͏ʹϞχλϦϯάΤʔ δΣϯτίϯςφΛσϓϩΠ • ؂ࢹର৅ͷίϯςφͷΈʹϑΥʔΧεͰ͖ɺίϯςφؒͰωοτϫʔ Ϋ΍ϑΝΠϧͳͲͷϦιʔεΛڞ༗Ͱ͖ΔͨΊ࣮૷͸γϯϓϧ • 1ϗετʹෳ਺ͷΤʔδΣϯτ͕σϓϩΠ͞ΕΔͷͰϦιʔε໘Ͱ೉ ͋Γ
38. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣ
39. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ)
40. MackerelίϯςφΤʔδΣϯτ(ύϒϦοΫϕʔλ) • ECS Task/Kubernetes Podઐ༻ͷϞχλϦϯάΤʔδΣϯτ • Sidecar Monitoring AgentΛ࠾༻ • কདྷతʹFargateͷΑ͏ͳΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφϓϥοτ ϑΥʔϜ͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • ܰྔΤʔδΣϯτͱͯ͠࠶ઃܭͯ͠ϦιʔεফඅΛܰݮ • ΫϥελΤʔδΣϯτͷAPI͔ΒϦιʔεϝτϦΫεΛऔಘ • SidecarʹΑΔϦιʔεڞ༗ʹΑΓ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹ͕Մೳ
41. ·ͱΊ
42. ·ͱΊ • ΞϓϦέʔγϣϯϝτϦΫεͱϦιʔεϝτϦΫεʹΘ͚ͯߟ͑Δ • ίϯςφͷϝτϦΫεͱͦͷऔಘํ๏Λ஌Δ • ίϯςφΫϥελʹ͓͚ΔϞχλϦϯάύλʔϯͱͯ͠per-Host Monitoring AgentͱSidecar Monitoring Agent͕͋Δ