Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prom...
Search
Manabu Matsuzaki
October 27, 2017
Technology
1
3.4k
ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus
モニタリング勉強会の発表資料です
Manabu Matsuzaki
October 27, 2017
Tweet
Share
More Decks by Manabu Matsuzaki
See All by Manabu Matsuzaki
Spring BootユーザのためのArmeria入門 #jsug / Introduce to Armeria for Spring users
matsumana
0
2.6k
Canary Release with Argo Rollouts #ふくばねてす / canary-release-with-argo-rollouts
matsumana
1
1.1k
Getting started Central Dogma with Golang #fukuokago #umedago / getting-started-central-dogma-with-golang
matsumana
0
860
Micrometer入門 #javaq / introduce-to-micrometer
matsumana
1
2.7k
ArmeriaとCentral Dogmaから学ぶ、マイクロサービスに必要な機能 #edayfuk / lean-from-armeria-and-central-dogma
matsumana
0
4.1k
SREcon19 Americas 参加レポート #srefukuoka / srecon19-americas-report
matsumana
0
830
SRE入門 & チームで取り組んでいるSRE #srefukuoka / introduce-to-sre
matsumana
0
1.2k
Introduce to Armeria and Central Dogma #GWD_Nulab / introduce-to-armeria-and-central-dogma
matsumana
0
550
Connector/JでMaster/Slave Replication構成のMySQLに接続する #mysql_casual_fukuoka /connector-j-master-slave-replication
matsumana
0
1.4k
Other Decks in Technology
See All in Technology
What to do after `laravel new`
mattstauffer
0
120
DynamoDBの"Replacement"時にデータが消されないようにCustom Resource Provider Frameworkでカスタムリソース作ってみた件
diggymo
0
130
君は隠しイベントを見つけれるか?
mujyun
0
390
マルチモーダルRAGやってみた
tanimon
0
110
LINEヤフー株式会社における音声言語情報処理AI研究開発@SP/SLP研究会 2024.10.22
lycorptech_jp
PRO
2
240
"君は見ているが観察していない"で考えるインシデントマネジメント
grimoh
3
530
Shift-from-React-to-Vue
calm1205
4
1.5k
スプリントゴールにチームの状態も設定する背景とその効果 / Team state in sprint goals why and impact
kakehashi
2
120
物価高なラスベガスでの過ごし方
zakky
0
490
Exadata Database Service on Cloud@Customer セキュリティ、ネットワーク、および管理について
oracle4engineer
PRO
0
1.1k
AWS CDKでデータリストアの運用、どのように設計する?~Aurora・EFSの実践事例を紹介~/aws-cdk-data-restore-aurora-efs
mhrtech
4
800
AWSコンテナ本出版から3年経った今、もし改めて執筆し直すなら / If I revise our container book
iselegant
18
4.2k
Featured
See All Featured
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
A Philosophy of Restraint
colly
203
16k
The Cost Of JavaScript in 2023
addyosmani
45
6.6k
How to train your dragon (web standard)
notwaldorf
88
5.7k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
46
2.1k
Six Lessons from altMBA
skipperchong
26
3.5k
Writing Fast Ruby
sferik
626
61k
Automating Front-end Workflow
addyosmani
1366
200k
Building an army of robots
kneath
302
42k
Measuring & Analyzing Core Web Vitals
bluesmoon
2
53
Product Roadmaps are Hard
iamctodd
PRO
48
10k
What's in a price? How to price your products and services
michaelherold
243
12k
Transcript
ϢʔβઢͰͷPrometheus ϞχλϦϯάษڧձ 2017/10/27 @matsumana
ࣗݾհ • ໊લɿ দ࡚ ֶ • ॴଐɿ LINE Fukuokaגࣜձࣾ ։ൃࣨ
• Twitterɿ @matsumana
ΞδΣϯμ • ࣮ࡍʹͲ͏͍ͬͨϝτϦΫεΛऔ͍ͬͯΔͷ͔ • exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ·ͱΊ ※࣌ؒͷ߹্ɺPrometheusೖతͳࣄ͠·ͤΜ
લఏ • ݱࡏ༻͍ͯ͠Δόʔδϣϯ • Prometheus 1.8.0
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • loading • CPU༻ • ϝϞϦ༻
• εϫοϓ • slab • context switch
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • node_exporter • σΟεΫI/O • σΟεΫ༻ • inode
• TCPଓ • ESTABLISHED • TIME_WAIT • ωοτϫʔΫ τϥϑΟοΫ
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • apache_exporter • mod_statusϞδϡʔϧͰऔΕΔϝτϦΫε • nginx_exporter • ngx_http_stub_status_moduleϞδϡʔϧͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • plack_exporter • Plack::Middleware::ServerStatus::LiteͰऔΕΔϝτϦΫε
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • JVMͷڞ௨తͳϝτϦΫε • Heap • Metaspace
• GC • Thread • Code cache
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • jmx_exporter • ϛυϧΣΞ/ϥΠϒϥϦݻ༗ͷϝτϦΫε • Tomcat • Jetty
• HikariCP • Caffeine • etc…
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • memcached_exporter • ΩϟογϡαΠζ • Ωϟογϡ • Ωϟογϡώοτ
• operation(get/set) • Eviction
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • mysqld_exporter • QPS • Slow query •
Replicaton delay • binlog • connection • InnoDBؔ࿈
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • redis_exporter • ༻ϝϞϦ • Key • Eviction
• Slow query
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Cluster health • Document(primary shardͷΈ)
• DocumentαΠζ(primary shardͷΈ) • Thread pool queue • Thread pool rejected • Circuit Breaker tripped
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Fielddata cache size • Query
cache size • Query cache hit,miss,total • Request cache size
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluent-plugin-prometheus • ֤BufferedOutput pluginͷϝτϦΫε • buffer_queue_length •
buffer_total_bytes • retry_count
ݱࡏ͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε ʢҰ෦ͷΈൈਮʣ • fluentd_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • td-agent_exporter
• fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻ • fluent-agent-lite_exporter • fluentdϓϩηε୯ҐͷCPU༻ • fluentdϓϩηε୯ҐͷϝϞϦ༻
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • https://prometheus.io/docs/instrumenting/exporters/ • https://github.com/prometheus/prometheus/wiki/Default- port-allocations • άάΔ • ͕ࣗඞཁͳϝτϦΫε͕export͞Ε͍ͯͳΕPR͢Δ
·ͣɺଞͷਓ͕࡞ͬͨͷ͕ແ͍͔୳ͯ͠ΈΔ
• ΦϑΟγϟϧͷΨΠυϥΠϯ https://prometheus.io/docs/instrumenting/writing_exporters/ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• ΦϑΟγϟϧͷclient library https://prometheus.io/docs/instrumenting/clientlibs/ • ΦϑΟγϟϧgolang, Java, Python, RubyͷΈ •
Third partyʹPHPͳͲ͋Δ exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
• exportergoͰॻ͘ͷ͕ྑͦ͞͏͚ͩͲҙ͋Δ • Linux Kernel 2.6.23ະຬgolangࣗମ͕αϙʔτͯ͠ͳ͍ CentOS5ܥͩͱgolangͷexporterਖ਼͘͠ಈ͔ͳ͍͔ https://github.com/golang/go/wiki/MinimumRequirements exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ࣗͰ࡞Δ߹
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ଞͷexporterͰΘΕͯͳ͍portΛ͏ɻ࡞ͬͨΒهࡌ͠Α͏ https://github.com/prometheus/prometheus/wiki/Default-port- allocations • ϝτϦΫε໊ɺϥϕϧ໊ͷϕετϓϥΫςΟεʹै͏ https://prometheus.io/docs/practices/naming/ •
exporterͰϝτϦΫεΛܭࢉ͠ͳ͍ɻPromQLͰܭࢉ͢Δ • ྫʣΩϟογϡώοτͳͲ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε • exporterͷόʔδϣϯ൪߸ΛϝτϦΫεͱͯ͠export͢Δ • ࢹରͷαʔόexporterͷ͕૿͑ΔͱɺͲͷόʔδϣ ϯ͕σϓϩΠ͞Ε͍ͯΔ͔Θ͔Βͳ͘ͳΔ • ͪͳΈʹɺPrometheusຊମҎԼͷΑ͏ͳϝτϦΫεΛ export͍ͯ͠Δ prometheus_build_info{branch=“master",goversion="go1.8.3",revision="3afb3fffa3a29
c3de865e1172fb740442e9d0133",version="1.7.1"} 1 ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ
exporterΛ࡞Δ࣌ͷϕετϓϥΫςΟε ΦϑΟγϟϧΨΠυϥΠϯʹهࡌ͞Ε͍ͯΔͷΛҰ෦͝հ • ̋̋̋_up ϝτϦΫεΛexport͢Δ • ྫ͑ɺMySQLͷ߹mysql_upͱ͍͏ϝτϦΫε • ࢹର͔ΒϝτϦΫεऔಘޭͳΒ1ɺࣦഊͳΒ0Λexport •
ཧ༝ޙड़
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ྫ1) exporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ྫ2) exporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
• ྫ3) dailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • Prometheusαʔό͕ࣗɺશͯͷexporterʹ͍ͭͯ upϝτϦΫεΛه͍ͯ͠Δ https://prometheus.io/docs/concepts/jobs_instances/ ྫʣnode_exporterͷ߹ up{instance="host00:9100", job="node"} 1
ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ALERT ExporterDown IF up == 0 FOR 1m LABELS
{severity="major"} ANNOTATIONS { description="{{ $labels.instance }} of job {{ $labels.job }} has been down” } ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ͳͷͰɺҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Ε શͯͷexporterʹରԠͰ͖Δ
Ξϥʔτϧʔϧͷॻ͖ํͷྫ ྫ̎ʣexporertͷࢹର͕མ͍ͪͯͳ͍͔Ͳ͏͔ࢹ͢Δ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ͢Εɺ ̋̋̋_upϝτϦΫεΛ࣋ͭશͯͷexporterʹରԠͰ͖Δ ALERT ExporterTargetDown IF {__name__=~".*_up"} ==
0 FOR 1m LABELS {severity="major"} ANNOTATIONS { summary="Exporter target ( {{ $labels.job }} ) down" }
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • node_exporterͷTextfile CollectorػೳΛͬͯϝτϦΫεΛexport͢Δ (pushgatewayΛͬͯϝτϦΫεexportग़དྷΔ) • ͜Μͳײ͡ͷshellΛjobεέδϡʔϥͰ࣮ߦ͢ΔΑ͏ʹͯ͠ɺ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ #!/bin/bash #
్தͰΤϥʔ͕ൃੜͨ͠Βऴྃ͢Δ set -e UNIXTIME=$(date +%s) RESULT=/path/to/node_exporter_textfile_directory/my_batch_job.prom # execute job /path/to/my_batch_job.sh # write metrics echo my_batch_job_completion_time{cron="my_batch_job", period="daily"} $UNIXTIME > $RESULT.tmp mv $RESULT.tmp $RESULT
Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛొ͢ΕݕՄೳ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔ࢹ͢Δ ALERT DailyCronLate IF time() - {period="daily"}
> 60 * 60 * 24 FOR 5m LABELS {severity="warning"} ANNOTATIONS { summary="Daily cronjob for {{ $labels.cron }} has not run in 24 hours” }
·ͱΊ • ͍ͬͯΔexporterͱɺऔಘग़དྷΔϝτϦΫεͷҰ෦Λ͝հ ͠·ͨ͠ • exporterΛ࡞Δ߹ͷҙ(golang)ʹ͍ͭͯ͝հ͠·ͨ͠ • exporterΛ࡞Δ߹ͷϕετϓϥΫςΟεΛ͍͔ͭ͘ൈਮͯ͠ ͝հ͠·ͨ͠ •
ΞϥʔτϧʔϧͷྫΛ͍͔ͭ͘͝հ͠·ͨ͠
Thank you :)