Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Me...

Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo

Hayato Imai

March 01, 2019
Tweet

More Decks by Hayato Imai

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 2017೥11݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़

    ίϯςφٕज़ೖ໳Ծ૝Խͱͷҧ͍Λ஌Γɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ 
  2. ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ •

    1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χ΢ϯτ • ʰγεςϜΛߏ੒͢Δ࠷খ୯Ґ(ͩͱγεςϜΛ؅ཧ͢Δ্ͰΈͳ ͢΂͖΋ͷ)ʱ • see. ʮFAQɾϗετ਺ͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
  3. ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO "1* ˔ ˔ ☓

    ☓ 5BTL.FUBEBUB &OEQPJOUW ☓ ☓ ˔ ˔ 5BTL.FUBEBUB &OEQPJOUW ˔ ˔ ˔ ☓
  4. Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARN΍εςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫ΍CPUɺϝϞϦLIMIT͸औಘͰ͖ͳ͍ • Docker

    stats API(docker.sock)΍cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ
  5. Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫ͸Docker stats APIͦͷ΋ͷΛฦ͢

    • v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍͸ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpc͸v3ʹҠߦ༧ఆ
  6. root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0":

    { "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3
  7. root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null

    e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
  8. -bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3))

    -bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2
  9. ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ৔߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ • EC2/Host, EC2/awsvpc,

    FargateͰ͸ɺίϯςφಉ࢜͸ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗ਎ͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠౤ߘ
  10. kubelet APIͷAuthN/AuthZ • kubelet portͰ͸ೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ •

    Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ
  11. ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰ͸read-only portΛར༻ • ઃఆͰkubelet portʹ੾Γସ͑Մೳ • τʔΫϯೝূΛαϙʔτ

    • "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍৔߹͸automountServiceAccountTokenઃఆ΋νΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ
  12. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups:

    [""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)
  13. apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io

    kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)
  14. ۙ೔ରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc

    • docker.sock΍cgroupfs΁ͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚ΢ϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍