Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Monitoring Docker with Mackerel (2015 ver.)
Search
Shinji Tanaka
March 29, 2015
Technology
300
1
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Monitoring Docker with Mackerel (2015 ver.)
クラウドごった煮 2015/3/28
Shinji Tanaka
March 29, 2015
Transcript
44 slides
Monitoring Docker with Mackerel 2015ฤ Ϋϥυͬͨࣽ͝ #11, 2015/03/28 Hatena
Co., Ltd. Shinji Tanaka @stanaka
Shinji Tanaka / ాத৻࢘ @stanaka / id:stanaka גࣜձࣾͯͳ ࣥߦһCTO 2
͋͋ ࠷ۙɺిࢠ൛ग़ͨΑ͏Ͱ͢
Agenda • Docker (cgroup) Statics • Host/Container Patterns • Monitoring
Docker with Mackerel
1 Docker (cgroup) Statistics
DockerͷϞχλϦϯάͷجຊ • cgroupͷ࡞๏ʹै͏ • ίϯςφଆ͔Βɺϗετશମ͔͠ ݟ͑ͳ͍ • /proc/loadavg, /proc/stat
• ݟ͍ͨ߹`-‐v`ͰऔΓࠐΉ
ϗετଆ͔ΒͷϞχλϦϯά • cgroupͷϨϙʔτΛར༻ • /sys/fs/cgroup/ {cpuacct,
memory} • ଞʹblkio, cpu, devicesͳͲͷα ϒγεςϜ͕͋Δ(֤छ੍ݶͳͲͷ੍ ޚ༻) VCVOUVͷ߹ LFSOFMHFOFSJDEPDLFS
CPU༻ • /sys/fs/cgroup/cpuacct/docker/{ίϯςφID}/cpuacct.stat ! ! ! • user … ϢʔβʔϞʔυͷCPU
࣌ؒ • system … γεςϜϞʔυͷCPU ࣌ؒ • CPU࣌ؒͷ୯Ґ: USER_HZ (σϑΥϧτ10ms) user 480 system 116
cpuacctͷଞͷࢦඪ • cpuacct.usage: ૯CPU࣌ؒ (φϊඵ) echo 0 > /cgroup/cpuacct/cpuacct.usage
ͰϦηοτ • cpuacct.usage_percpu: ֤CPUͷCPU ࣌ؒ (φϊඵ)
ϝϞϦ༻ • /sys/fs/cgroup/memory/docker/{ίϯςφID}/memory.stat cache 761856 rss 25681920 rss_huge
0 mapped_file 16384 pgpgin 12569 … total_cache 761856 total_rss 25681920 … ͍ΘΏΔ344STT NBQQFE@pMF
memory.statৄࡉ • cache • rss • mapped_file
• pgpgin • pgpgout • swap • active_anon • inactive_anon • active_file • inactive_file • unevictable • hierarchical_memory _limit • hierarchical_memsw • total_ࢠάϧʔϓΛ ؚΉ ৄࡉIUUQTXXXLFSOFMPSHEPD%PDVNFOUBUJPODHSPVQTNFNPSZUYU
memoryؔ࿈(ͦͷଞ) memory.usage_in_bytes ϝϞϦ༻ྔ memory.memsw.usage_in_bytes ϝϞϦ༻ྔͱεϫοϓྖҬͷ memory.max_usage_in_bytes ϝϞϦ࠷େ༻ྔ memory.memsw.max_usage_in_bytes εϫοϓ࠷େ༻ྔ memory.limit_in_bytes
ϝϞϦͷ੍ݶ memory.memsw.limit_in_bytes ϝϞϦͱεϫοϓ੍ݶ memory.failcnt ϝϞϦ੍ݶʹୡͨ͠ճ memory.memsw.failcnt ϝϞϦ੍ݶʹୡͨ͠ճ
ڞ௨Ϩϙʔτ • tasks ࣮ߦதͷϓϩηεͷҰཡ(PID) • cgroup.procs ࣮ߦதͷεϨουάϧʔϓͷҰཡ(TGID) % cat /sys/fs/cgroup/memory/docker/…./cgroup.procs
31218 31307 % ps aux | egrep '(31218|31307)' root 31218 0.0 0.0 4408 164 ? Ss 13:15 0:00 /bin/sh -‐ c rackup -‐p4567 root 31307 0.0 5.2 281364 26336 ? Sl 13:15 0:06 ruby1.9.1 /usr/local/bin/rackup -‐p4567
2 Host/Container Patterns
ϗετͱίϯςφͷؔ • 1:n • CIαʔόɾ։ൃαʔόͳͲ • 1:1 • ϓϩμΫγϣϯڥͳͲ
Pattern 1:n • CIαʔόɾ։ൃαʔό • ίϯςφԾԽڥΛଟಈ࡞ • ಠཱͨ͠ڥ • ىಈఀࢭ͕ߴ
• Ϧιʔεͷڞ༗ʹΑΔޮԽ
Pattern 1:n • CIαʔόɾ։ൃαʔό )PTU "QQ qVFOUE NBDLFSFM BHFOU "QQ
+FOLJOT
Pattern 1:1 • ίϯςφԾڥΛҰͭͷΈಈ࡞ • ϦιʔεΛಠ • ڥߏங໘ʹಛԽ • ϓϩμΫγϣϯڥʹ߹͏
• AWS BeanstalkͷDocker support
Pattern 1:1 • ϓϩμΫγϣϯαʔό )PTU qVFOUE NBDLFSFM BHFOU "QQ
3 Monitoring Docker with Mackerel
Mackerel https://mackerel.io/
ಛ • ʮׂʯʹΑΔϗετ܈ͷཧ • ΤʔδΣϯτʹΑΔϦιʔεࢹ ϩ ʔ ϧ
ϩʔϧຖͷάϥϑͰՄࢹԽ
mackerel-agent • Go Ͱॻ͔ΕͨެࣜΤʔδΣϯτ • Linux αϙʔτ (CentOS 5/6, Debian
6/7) • ϗετεϖοΫͱϝτϦοΫΛऩू mackerelio/mackerel-agent
Blue-green DeploymentରԠ • ϩʔϧ୯ҐͰͷՄࢹԽ • ୀࡁΈίϯςφͷάϥϑอ࣋ Deployment
ΦʔτεέʔϧରԠ • αʔό͕ॖୀͯ͠άϥϑΔ
Raspberry Pi Ͱಈ͖·͢ • Թɾ࣪ηϯα → Slack௨
Docker with Mackerel
mackerel/mackerel-agent on Docker Hub https://registry.hub.docker.com/u/stanaka/mackerel-‐agent/
How to Use • ίϚϯυҰൃ • ඞཁͳύεɾAPIKEYΛ͢ docker run -‐h
`hostname` \ -‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=<APIKEY>‘ \ mackerel/mackerel-‐agent
DockerFile(ൈਮ) FROM debian:jessie ! # setup mackerel-‐agent RUN
apt-‐get update \ .. && apt-‐get -‐y install mackerel-‐agent \ .. ! # setup docker plugin COPY mackerel-‐plugin-‐docker.rb /mackerel-‐plugin-‐docker.rb RUN echo [plugin.metrics.docker] \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf RUN echo command = \"ruby /mackerel-‐plugin-‐docker.rb\" \ >> /etc/mackerel-‐agent/mackerel-‐agent.conf ! # boot mackerel-‐agent CMD /startup.sh
֤ίϯςφͷใΛऩू )PTU qVFOUE NBDLFSFM BHFOU "QQ
༷ࢠ(ίϯςφຖͷCPU)
༷ࢠ(ίϯςφຖͷϝϞϦ)
༷ࢠ(·ͱΊͯ)
ίϯςφͷதͷ ใΛऩू͍ͨ͠
ίϯςφͷதͷใΛऩू )PTU NBDLFSFM BHFOU "QQ
Memcachedͷྫ % docker run -‐d -‐P \
-‐-‐name memcached -‐p 11211:11211 \ sylvainlasnier/memcached )PTU NBDLFSFM BHFOU NFNDBDIFE
linkΦϓγϣϯͰଓ % docker run -‐i -‐t -‐h `hostname` \
-‐v /var/run/docker.sock:/var/run/docker.sock \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -‐e 'apikey=<APIKEY>' -‐-‐link memcached:memcached \ mackerel-‐agent /bin/bash ! root@dev:/# env HOSTNAME=develop04.host.h MEMCACHED_PORT_11211_TCP_PORT=11211 MEMCACHED_PORT_11211_TCP_PROTO=tcp MEMCACHED_NAME=/tender_curie/memcached MEMCACHED_PORT=tcp://172.17.0.88:11211 MEMCACHED_PORT_11211_TCP_ADDR=172.17.0.88 MEMCACHED_PORT_11211_TCP=tcp://172.17.0.88:11211
ڥมͰΞυϨεࢦఆ root@dev:/# /usr/local/bin/mackerel-‐plugin-‐memcached \ -‐host=$MEMCACHED_PORT_11211_TCP_ADDR memcached.connections.curr_connections
10 1427507503 memcached.cmd.cmd_get 0 1427507503 memcached.cmd.cmd_set 0 1427507503
ϓϥάΠϯઃఆΛࠐΉ % docker run -‐h `hostname` \ -‐v
/var/run/docker.sock:/var/run/docker.sock \ -‐v /var/lib/mackerel-‐agent/:/var/lib/mackerel-‐agent/ \ -‐v /proc/mounts:/host/proc/mounts:ro \ -‐v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro -‐e 'apikey=<APIKEY>‘ \ -‐-‐link memcached:memcached \ -‐v /etc/mackerel-‐agent/conf.d:/etc/mackerel-‐agent/conf.d:ro -‐e ‘include=/etc/mackerel-‐agent/conf.d/*' \ mackerel/mackerel-‐agent % cat /etc/mackerel-‐agent/conf.d/memcached.conf [plugin.metrics.memcached] command = "/usr/local/bin/mackerel-‐plugin-‐memcached -‐ host=$MEMCACHED_PORT_11211_TCP_ADDR"
MemcachedͷϝτϦοΫ
·ͱΊ • AgentΛίϯςφͰཱͯΔͱ؆୯ • ॏ͍ίϯςφ͕͙͢ʹൃݟͰ͖Δ • ֤ίϯςφͷதݟΕΔ
Q & A
More Decks by Shinji Tanaka
See All by Shinji Tanaka
Mackerel loves Machine Learnings
stanaka
0
3.3k
How to Contribute Go
stanaka
4
19k
Introduction to Mackerel
stanaka
0
1.2k
Upcoming features in 2016, Mackerel
stanaka
0
3.6k
Mackerel's first year and onward
stanaka
0
3.3k
Mackerel's first year and onward (in English)
stanaka
0
500
Blue-Green Deployment with ECS and monitoring
stanaka
0
680
Mackerel Meetup #4 (in Eng)
stanaka
0
680
Mackerel Meetup #4
stanaka
0
3.5k
Other Decks in Technology
See All in Technology
HTTP通信に使おうとすると実はめんどくさいfile_get_contents
maguroalternative
0
220
[NW-JAWS DAY]「うちのAWS、インターネット出られないんで」― AWS外の自社サービスをPrivateLinkでシュッと公開する方法
ykimi
3
880
落とし穴にハマってわかった 開発生産性を測るときに気をつけたいこと
munakata
3
420
SORACOM × AWS IoT Core でSREする
butadora
0
110
こんなに変わった! AWSネットワークの常識
shotashiratori
16
11k
AI対話型分析を作成し、リリースするまでの話 / The Road to Release: Buiding AI Conversational Analytics
yappli_developers
0
170
常時稼働エージェントの作り方(Claude公式ブログを資料化)
minorun365
PRO
16
28k
CloudTrail を推したい
kazzpapa3
1
140
Biohub - Cell Tracking During Development 5th place solution
rist
0
180
人間+AIを「チーム」としてプロセスを再設計する
littlehands
4
3.5k
2026-11-16 JAWS-UG コンテナ支部 #31 ゆるLT大会 ECR のライフサイクルポリシーを いい感じに設定したい / ECR Lifecycle policy
masasuzu
3
550
速くなるほど、深く考える AI時代のQAに学ぶ
kobayashiyorimitsu
0
270
Featured
See All Featured
Building Experiences: Design Systems, User Experience, and Full Site Editing
marktimemedia
1
670
Six Lessons from altMBA
skipperchong
29
4.6k
Building Flexible Design Systems
yeseniaperezcruz
330
41k
The agentic SEO stack - context over prompts
schlessera
0
1k
From π to Pie charts
rasagy
1
450
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
360
4 Signs Your Business is Dying
shpigford
187
23k
Collaborative Software Design: How to facilitate domain modelling decisions
baasie
1
380
The Art of Programming - Codeland 2020
erikaheidi
57
15k
Automating Front-end Workflow
addyosmani
1370
210k
Visualization
eitanlees
153
17k
State of Search Keynote: SEO is Dead Long Live SEO
ryanjones
0
350