Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

2019/03/01 https://mackerelio.connpass.com/event/118995/

Transcript

1. MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφ؂ࢹʹ͍ͭͯ .BDLFSFM
   .FFUVQ
   
   JEIBZBKP@
2. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 2017೥11݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़ ίϯςφٕज़ೖ໳
   
   Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏
   IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ 
3. ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷ঺հ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ
4. MackerelίϯςφΤʔδΣϯτͷ঺հ
5. MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218
6. ύϒϦοΫϕʔλͱ͸ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂ஌ͷ্Ͱඇޓ׵मਖ਼Λ࣮ࢪ
7. OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent • Issue΍PR͸ݪଇӳޠͰ͓ئ͍͠·͢
8. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ • 1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ ͢΂͖΋ͷ)ʱ • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
9. αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) • Kubernetes • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ
10. Web UI/ϗετҰཡ
11. Web UI/ϗετৄࡉ
12. Web UI/ϩʔϧάϥϑ
13. ΍Δ͜ͱ • λεΫ/Podͷ͢΂ͯͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠౤ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓ౰ͯ • ϓϥάΠϯͷར༻ • ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ
14. ΍Βͳ͍͜ͱ • Ϋϥελ΍ϊʔυͷϝτϦοΫ͸ऩू͠ͳ͍ • ϊʔυ਺ɺλεΫ/Pod਺, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻ • αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷ؂ࢹ • Docker͸ʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ
15. αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷ؅ཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ؂ࢹର৅ͷΞϓϦέʔγϣϯ࢓༷ʹ͍ۙ؂ࢹઃఆ͕Մೳ
16. ϝτϦοΫऔಘઓུղઆ
17. ϝτϦοΫऔಘΞʔΩςΫνϟ
18. ECS/Fargate
19. ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO
    "1* ˔ ˔ ☓ ☓ 5BTL
    .FUBEBUB
    &OEQPJOU
    W ☓ ☓ ˔ ˔ 5BTL
    .FUBEBUB
    &OEQPJOU
    W ˔ ˔ ˔ ☓
20. Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍ • Docker stats API(docker.sock)΍cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ
21. Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢ • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpc͸v3ʹҠߦ༧ఆ
22. ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖Δͷ͸EC2/Bridge͚ͩ...... • Docker stats API(libnetwork)͸bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮૷ʹৄ͍͠ํɺͥͻ࠙਌ձͰ͓࿩͠·͠ΐ͏ʂ
23. root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0": { "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3
24. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
25. -bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3)) -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2
26. EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫ͸λεΫͷίϯςφ͝ͱʹҟͳΔ
27. EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
28. EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
29. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ • EC2/Host, EC2/awsvpc, FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
30. • ಉ͡ίϯςφఆٛͰ΋ɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD
    'BSHBUF
31. • ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓ͸Task Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
32. Kubernetes
33. KubernetesͷCore metrics pipeline
34. kubelet API • Podͷϝλσʔλ΍ϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN • read-only port͕ແޮͳ؀ڥ΋͋Δ
35. kubelet APIͷAuthN/AuthZ • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ • Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ
36. ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰ͸read-only portΛར༻ • ઃఆͰkubelet portʹ੾Γସ͑Մೳ • τʔΫϯೝূΛαϙʔτ • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ
37. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups: [""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)
38. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)
39. • kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτ͸read-only PortΛར༻ • kubelet portͰ͸τʔΫϯೝূʹରԠ • ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
40. ࠓޙͷίϯςφΤʔδΣϯτ
41. ۙ೔ରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍
42. ల๬ͱߏ૝ • ϓϥάΠϯར༻ͷརศੑ޲্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀ໾ޙɺҰఆظؒܦաͰඇදࣔͱͳΔ • Prometheus΍ServiceMeshͱͷ࿈ܞ
43. ·ͱΊ
44. ·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛ؂ࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛ؂ࢹ͢Δ - Mackerel ϔϧϓʯ
45. ϑΟʔυόοΫΛ͓଴͍ͪͯ͠·͢