Upgrade to Pro — share decks privately, control downloads, hide ads and more …

ユーザ目線でのPrometheus #mackerel_ug /monitoring-prom...

ユーザ目線でのPrometheus #mackerel_ug /monitoring-prometheus

モニタリング勉強会の発表資料です

Manabu Matsuzaki

October 27, 2017
Tweet

More Decks by Manabu Matsuzaki

Other Decks in Technology

Transcript

  1. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε
 ʢҰ෦ͷΈൈਮʣ • elasticsearch_exporter • Cluster health • Document਺(primary shardͷΈ)

    • DocumentαΠζ(primary shardͷΈ) • Thread pool queue਺ • Thread pool rejected਺ • Circuit Breaker tripped਺
  2. ݱࡏ࢖͍ͬͯΔexporterͱɺऔಘ͍ͯ͠ΔϝτϦΫε
 ʢҰ෦ͷΈൈਮʣ • fluentd_exporter • fluentdϓϩηε୯ҐͷCPU࢖༻཰ • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰ • td-agent_exporter

    • fluentdϓϩηε୯ҐͷCPU࢖༻཰ • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰ • fluent-agent-lite_exporter • fluentdϓϩηε୯ҐͷCPU࢖༻཰ • fluentdϓϩηε୯ҐͷϝϞϦ࢖༻཰
  3. Ξϥʔτϧʔϧͷॻ͖ํͷྫ ALERT ExporterDown IF up == 0 FOR 1m LABELS

    {severity="major"} ANNOTATIONS { description="{{ $labels.instance }} of job {{ $labels.job }} has been down” } ྫ1ʣexporter͕མ͍ͪͯͳ͍͔Ͳ͏͔؂ࢹ͢Δ • ͳͷͰɺҎԼͷ༷ͳΞϥʔτϧʔϧΛ1ͭొ࿥͢Ε͹
 શͯͷexporterʹରԠͰ͖Δ
  4. Ξϥʔτϧʔϧͷॻ͖ํͷྫ • node_exporterͷTextfile CollectorػೳΛ࢖ͬͯϝτϦΫεΛexport͢Δ
 (pushgatewayΛ࢖ͬͯ΋ϝτϦΫε͸exportग़དྷΔ) • ͜Μͳײ͡ͷshellΛjobεέδϡʔϥͰ࣮ߦ͢ΔΑ͏ʹͯ͠ɺ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔؂ࢹ͢Δ #!/bin/bash #

    ్தͰΤϥʔ͕ൃੜͨ͠Βऴྃ͢Δ set -e UNIXTIME=$(date +%s) RESULT=/path/to/node_exporter_textfile_directory/my_batch_job.prom # execute job /path/to/my_batch_job.sh # write metrics echo my_batch_job_completion_time{cron="my_batch_job", period="daily"} $UNIXTIME > $RESULT.tmp mv $RESULT.tmp $RESULT
  5. Ξϥʔτϧʔϧͷॻ͖ํͷྫ • ҎԼͷ༷ͳΞϥʔτϧʔϧΛొ࿥͢Ε͹ݕ஌Մೳ ྫ̏ʣdailyόον͕ਖ਼ৗʹಈ͍͍ͯΔ͔Ͳ͏͔؂ࢹ͢Δ ALERT DailyCronLate IF time() - {period="daily"}

    > 60 * 60 * 24 FOR 5m LABELS {severity="warning"} ANNOTATIONS { summary="Daily cronjob for {{ $labels.cron }} has not run in 24 hours” }