Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Me...
Search
Hayato Imai
March 01, 2019
Technology
1
8.7k
Mackerelコンテナエージェントによる コンテナ監視について / Mackerel Meetup #13 Tokyo
2019/03/01
https://mackerelio.connpass.com/event/118995/
Hayato Imai
March 01, 2019
Tweet
Share
More Decks by Hayato Imai
See All by Hayato Imai
Kubernetes撤退、 その後のはてなの取り組み / kubernetes meetup tokyo number 52
hayajo
9
7.1k
Mackerelにおける Cloud Nativeへの取り組みと チームへ与えた変化 / CloudNative Days Tokyo 2020
hayajo
2
1.4k
MackerelにおけるKubernetes利用の取組みとこれから / Kubernetes Meetup Tokyo #22
hayajo
20
9.5k
Mackerelチームのコンテナ開発における戦略とこれから / 190722-cndt2019
hayajo
1
1.8k
AWSコンテナサービス入門 / nds60-jaws-ug
hayajo
0
2.8k
コンテナのメトリクスと モニタリングパターン / 190320-sakura-event
hayajo
6
1.6k
Docker for Mac/Windows ではじめる Kubernetes / NDS55 Docker with Kubernetes
hayajo
16
14k
Terrafromで構築するマルチクラウドプラットフォームインフラストラクチャ / NDS53 Terraform
hayajo
0
390
Ncatをつかおう / Use Ncat
hayajo
1
3.6k
Other Decks in Technology
See All in Technology
Vueで Webコンポーネントを作って Reactで使う / 20241030-cloudsign-vuefes_after_night
bengo4com
4
2.5k
GitHub Universe: Evaluating RAG apps in GitHub Actions
pamelafox
0
170
で、ValhallaのValue Classってどうなったの?
skrb
1
660
Commitment vs Harrisonism - Keynote for Scrum Niseko 2024
miholovesq
6
1k
Fargateを使った研修の話
takesection
0
110
Jr. Championsになって、強く連携しながらAWSをもっと使いたい!~AWSに対する期待と行動~
amixedcolor
0
180
小規模に始めるデータメッシュとデータガバナンスの実践
kimujun
3
560
新卒1年目が挑む!生成AI × マルチエージェントで実現する次世代オンボーディング / operation-ai-onboarding
cyberagentdevelopers
PRO
1
160
AIを駆使したゲーム開発戦略: 新設AI組織の取り組み / sge-ai-strategy
cyberagentdevelopers
PRO
1
130
失敗しないOpenJDKの非互換調査
tabatad
0
270
MAMを軸とした動画ハンドリングにおけるAI活用前提の整備と次世代ビジョン / abema-ai-mam
cyberagentdevelopers
PRO
1
110
現地でMeet Upをやる場合の注意点〜反省点を添えて〜
shotashiratori
0
500
Featured
See All Featured
Building an army of robots
kneath
302
42k
Why Our Code Smells
bkeepers
PRO
334
57k
Designing Experiences People Love
moore
138
23k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
231
17k
Teambox: Starting and Learning
jrom
132
8.7k
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
504
140k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
32
1.8k
The World Runs on Bad Software
bkeepers
PRO
65
11k
The Language of Interfaces
destraynor
154
24k
Learning to Love Humans: Emotional Interface Design
aarron
272
40k
The Cult of Friendly URLs
andyhume
78
6k
Practical Orchestrator
shlominoach
186
10k
Transcript
MackerelίϯςφΤʔδΣϯτʹΑΔ ίϯςφࢹʹ͍ͭͯ .BDLFSFM.FFUVQ JEIBZBKP@
ࣗݾհ • ࠓҪ൏ਓ(id:hayajo_77) • MackerelνʔϜ SRE • 201711݄ೖࣾ • ίϯςφཁૉٕज़ɺपลٕज़
ίϯςφٕज़ೖԾԽͱͷҧ͍ΛΓɺཁૉٕज़Λ৮ֶͬͯ΅͏ IUUQTFNQMPZNFOUFOKBQBODPNFOHJOFFSIVCFOUSZ
ΞδΣϯμ • MackerelίϯςφΤʔδΣϯτͷհ • ϝτϦοΫऔಘઓུղઆ • ࠓޙͷίϯςφΤʔδΣϯτ • ·ͱΊ
MackerelίϯςφΤʔδΣϯτͷհ
MackerelίϯςφΤʔδΣϯτ ʢύϒϦοΫϕʔλʣΛެ։͠·ͨ͠ https://mackerel.io/ja/blog/entry/weekly/20190218
ύϒϦοΫϕʔλͱ • কདྷతʹਖ਼ࣜͳαʔϏεϦϦʔεΛߦ͏༧ఆ͕͋ΔػೳΛઌߦͯ͠ ެ։ • ׆ൃʹมߋΛߦ͍ɺ৽͍͠όʔδϣϯΛఏڙ • ࣄલࠂͷ্Ͱඇޓमਖ਼Λ࣮ࢪ
OSSͱͯ͠ެ։ • DockerHub • mackerel/mackerel-container-agent • GitHub • mackerelio/mackerel-container-agent •
IssuePRݪଇӳޠͰ͓ئ͍͠·͢
ίϯςφઐ༻ͷܰྔΤʔδΣϯτ • ίϯςφઐ༻ͱͯ͠࠶ઃܭ • Amazon ECS, AWS Fargate, KubernetesʹରԠ •
1λεΫ/Podʹ͖ͭ1ϗετͱͯ͠Χϯτ • ʰγεςϜΛߏ͢Δ࠷খ୯Ґ(ͩͱγεςϜΛཧ͢Δ্ͰΈͳ ͖͢ͷ)ʱ • see. ʮFAQɾϗετͷܭࢉํ๏ʹ͍ͭͯ - Mackerel ϔϧϓʯ
αΠυΧʔίϯςφͱͯ͠σϓϩΠ • ϓϥοτϑΥʔϜʹΑͬͯઃఆ͕ҟͳΔ • ECS(EC2/Bridge, EC2/Host) • ECS(EC2/awsvpc, Fargate) •
Kubernetes • see. ʮίϯςφΛࢹ͢Δ - Mackerel ϔϧϓʯ
Web UI/ϗετҰཡ
Web UI/ϗετৄࡉ
Web UI/ϩʔϧάϥϑ
Δ͜ͱ • λεΫ/Podͷͯ͢ͷίϯςφͷϝτϦοΫΛγεςϜϝτϦοΫ ͱͯ͠ߘ • CPUɺϝϞϦɺωοτϫʔΫ • αʔϏε/ϩʔϧͷׂΓͯ • ϓϥάΠϯͷར༻
• ϓϥάΠϯΛΠϯετʔϧͨ͠DockerΠϝʔδͷ४උ͕ඞཁ
Βͳ͍͜ͱ • ΫϥελϊʔυͷϝτϦοΫऩू͠ͳ͍ • ϊʔυɺλεΫ/Pod, etc. • ϊʔυͷCPUɺϝϞϦɺωοτϫʔΫɺσΟεΫͳͲ • mackerel-agentΛར༻
• αϙʔτ͢ΔϓϥοτϑΥʔϜҎ֎ͷίϯςφͷࢹ • DockerʮDockerΛϞχλϦϯά͢Δ - Mackerel ϔϧϓʯ
αΠυΧʔύλʔϯͷ࠾༻ • FargateͷΑ͏ͳɺΠϯϑϥετϥΫνϟͷཧ͕ෆཁͳίϯςφ ར༻͕ओྲྀʹͳΔͱߟ͍͑ͯΔ • Mackerelʹ͓͚ΔϗετͷఆٛͱλεΫ/PodͷϥΠϑαΠΫϧ • ίϯςφؒͷϦιʔεڞ༗ • ࢹରͷΞϓϦέʔγϣϯ༷ʹ͍ۙࢹઃఆ͕Մೳ
ϝτϦοΫऔಘઓུղઆ
ϝτϦοΫऔಘΞʔΩςΫνϟ
ECS/Fargate
ECS/FargateͷAPI &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF *OUSPTQFDUJPO "1* ˔ ˔ ☓
☓ 5BTL.FUBEBUB &OEQPJOUW ☓ ☓ ˔ ˔ 5BTL.FUBEBUB &OEQPJOUW ˔ ˔ ˔ ☓
Introspection API • λεΫͷϝλσʔλΛฦ͢ • λεΫͷARNεςʔλεɺ֤ίϯςφͷDockerIDͳͲ • ίϯςφͷϝτϦοΫCPUɺϝϞϦLIMITऔಘͰ͖ͳ͍ • Docker
stats API(docker.sock)cgroupfsͰΧόʔ • EC2/Bridge, EC2/HostͰར༻ • Task Metadata Endpoint v3ʹରԠ༧ఆ
Task Metadata Endpoint v2/v3 • λεΫϝλσʔλɺίϯςφϝτϦοΫΛฦ͢ • ϝτϦοΫDocker stats APIͦͷͷΛฦ͢
• v2/v3ͰऔಘͰ͖Δσʔλʹେ͖ͳҧ͍ͳ͍ • EC2/awsvpc, FargateͰv2Λར༻ • EC2/awsvpcv3ʹҠߦ༧ఆ
ECS/FargateͷωοτϫʔΫϝτϦοΫ • ωοτϫʔΫϝτϦοΫ͕औಘͰ͖ΔͷEC2/Bridge͚ͩ...... • Docker stats API(libnetwork)bridgeϞʔυͷͱ͖͔͠ϝτϦοΫ ͕औΕͳ͍ͬΆ͍ • ࣮ʹৄ͍͠ํɺͥͻ࠙ձͰ͓͠·͠ΐ͏ʂ
root@ebb5c8c90634:/# curl -s $ {ECS_CONTAINER_METADATA_URI}/stats | jq .networks { "eth0":
{ "rx_bytes": 17331985, "tx_packets": 932, "rx_packets": 1353, "tx_bytes": 77755 } } e.g. EC2/Bridge w/ TMEv3
root@ip-10-0-10-144:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
e.g. EC2/Host, EC2/awsvpc w/ TMEv3 root@9ea93ec5d92b:/# curl -s ${ECS_CONTAINER_METADATA_URI}/stats | \ > jq .networks null
-bash-4.2# CID=$(basename $(head -n1 /proc/self/ cgroup | cut -d: -f3))
-bash-4.2# curl -s 169.254.170.2/v2/stats/${CID} | jq .networks null e.g. Fargate w/ TMEv2
EC2/BridgeͷωοτϫʔΫελοΫ • ωοτϫʔΫελοΫλεΫͷίϯςφ͝ͱʹҟͳΔ
EC2/HostωοτϫʔΫελοΫ • ϗετͷωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
EC2/awsvpc, FargateωοτϫʔΫελοΫ • ωοτϫʔΫελοΫΛλεΫͷίϯςφؒͰڞ༗
ίϯςφΤʔδΣϯτʹ͓͚ΔECS/Fargateͷ ωοτϫʔΫϝτϦοΫͷऔಘ • EC2/BridgeϞʔυͷ߹ɺDocker stats API͔Β֤ίϯςφͷωο τϫʔΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠ߘ • EC2/Host, EC2/awsvpc,
FargateͰɺίϯςφಉ࢜ωοτϫʔ ΫελοΫΛڞ༗͢ΔͷͰɺίϯςφΤʔδΣϯτࣗͷωοτϫʔ ΫϝτϦοΫΛऔಘͯ͠λεΫͷϝτϦοΫͱͯ͠ߘ
• ಉ͡ίϯςφఆٛͰɺىಈλΠϓɺωοτϫʔΫϞʔυͰΠϯλʔ ϑΣʔεͷݟ͑ํ͕ҟͳΔ λεΫͷInterfaceάϥϑ &$#SJEHF &$)PTU &$BXTWQD 'BSHBUF
• ىಈλΠϓɺωοτϫʔΫϞʔυʹΑͬͯར༻Ͱ͖ΔAPI͕ҟͳΔ • EC2ىಈλΠϓTask Metadata Endpoint v3ʹରԠத • ωοτϫʔΫϞʔυʹΑͬͯωοτϫʔΫϝτϦοΫͷऔಘํ๏͕ ҟͳΔ
ECS/Fargateʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
Kubernetes
KubernetesͷCore metrics pipeline
kubelet API • PodͷϝλσʔλϝτϦοΫɺϩάΛऔಘɺίϚϯυͷϩʔΧϧ ࣮ߦͷͨΊͷAPI • kubelet port(10250/HTTPS)ͱread-only port(10255/HTTP)Ͱ LISTEN
• read-only port͕ແޮͳڥ͋Δ
kubelet APIͷAuthN/AuthZ • kubelet portͰೝূ/ೝՄΛઃఆͰ͖Δ • Authentication • ಗ໊ΞΫηεɺΫϥΠΞϯτূ໌ॻೝূɺτʔΫϯೝূ •
Authorization • AlwaysAllow, Webhook • SubjectAccessReview APIʹΑΔݖݶνΣοΫ • see. ʮKubelet authentication/authorization - Kubernetesʯ
ίϯςφΤʔδΣϯτͱkubelet API • σϑΥϧτͰread-only portΛར༻ • ઃఆͰkubelet portʹΓସ͑Մೳ • τʔΫϯೝূΛαϙʔτ
• "nodes/proxy", "nodes/stats", "nodes/spec"ʹgetΞΫηε • ͏·͍͔͘ͳ͍߹automountServiceAccountTokenઃఆνΣοΫ • see. ʮKubernetesʹmackerel-container-agentΛηοτΞοϓ͢Δ - Mackerel ϔϧϓʯ
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: mackerel-container-agent rules: - apiGroups:
[""] resources: ["nodes/proxy", "nodes/stats", "nodes/spec"] verbs: ["get"] e.g. RBACઃఆ(ClusterRole)
apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: mackerel-container-agent-binding roleRef: apiGroup: rbac.authorization.k8s.io
kind: ClusterRole name: mackerel-container-agent subjects: - kind: ServiceAccount name: my-service-account namespace: default e.g. RBACઃఆ(ClusterRoleBinding)
• kubelet API͔ΒϝτϦοΫΛऔಘ͢Δ • σϑΥϧτread-only PortΛར༻ • kubelet portͰτʔΫϯೝূʹରԠ •
ඞཁʹԠͯ͡RBACΛઃఆ Kubernetesʹ͓͚ΔϝτϦοΫऔಘ·ͱΊ
ࠓޙͷίϯςφΤʔδΣϯτ
ۙରԠ༧ఆ • Task Metadata Endpoint v3ରԠ • EC2/Bridge, EC2/Host, EC2/awspvc
• docker.sockcgroupfsͷґଘΛͳ͘͢ • rootϢʔβඞཁͳ͠ • cgroupfsͷϚϯτϙΠϯτͷҧ͍Λؾʹ͠ͳͯ͘Α͍
లͱߏ • ϓϥάΠϯར༻ͷརศੑ্ • ϓϥάΠϯಉࠝΠϝʔδͷఏڙͳͲ • ΧελϜϝτϦοΫͷѻ͍ • ϗετಉ༷ʹୀޙɺҰఆظؒܦաͰඇදࣔͱͳΔ •
PrometheusServiceMeshͱͷ࿈ܞ
·ͱΊ
·ͱΊ • ίϯςφΤʔδΣϯτʢύϒϦοΫϕʔλʣΛϦϦʔε • λεΫ/PodΛࢹ͢ΔαΠυΧʔίϯςφ • ϓϥοτϑΥʔϜ͕ఏڙ͢ΔAPI͔ΒϝτϦοΫΛऩू • see. ʮίϯςφΛࢹ͢Δ
- Mackerel ϔϧϓʯ
ϑΟʔυόοΫΛ͓͍ͪͯ͠·͢
None