最近の監視(仮)/Recent system monitoring with mackerel

mackerel day #2

Transcript

1. ࠷ۙͷ؂ࢹ(Ծ) Mackerel Day #2 kazeburo
2. Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ • Ұ൪޷͖ͳ mackerelϓϥάΠϯ͸ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ໾ ʹཱͭͱ͜ΖͰ͸ diff-detector(ࣗ࡞)
3. https://ihara2525.tumblr.com/ “Ұߦͷϩάͷ޲͜͏ʹ͸ɺҰਓͷϢʔβ͕͍Δ”
4. • Τϥʔ͸όδΣοτͱ͍͏ߟ͑ • ࢒೦ʹࢥ͏ϢʔβΛগͳ͍͖͍ͯͨ͘͠ • ؂ࢹ͸Ϣʔβͷظ଴ʹԠ͑ΔͨΊͷπʔϧ
5. • ʮ؂ࢹʯʹΑΓ • ো֐Λະવʹ๷͙ • ো֐ʹૉૣ͘ؾͮ͘ • ো֐ͷݪҼಛఆͷώϯτ/MTTRͷ୹ॖ
6. mackerel-plugin-axslog ͱ೿ੜ
8. • mackerel-plugin-axslog • https://github.com/kazeburo/mackerel-plugin-axslog • mackerel-plugin-accesslog ͱಉ͘͡accesslogͷՄࢹԽ • status͝ͱͷ਺ͱׂ߹ɺresponse timeͷpercentile஋Λܭଌ • ूܭ͢ΔlabelΛมߋՄೳɾJSONαϙʔτ • ߴ଎
9. 0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s 7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ
10. • ߴ଎Խͷར఺ • γεςϜෛՙΛؾʹͤͣɺʮΨϯΨϯʯϝτϦΫεΛ૿΍ͤΔ • ϑΟϧλػೳͷ௥Ճ
11. [plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \
 --logfile /var/log/nginx/access.log --filter Googlebot" [plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \
 --logfile /var/log/nginx/access.log --filter AdsBot-Google"
12. • mackerel-plugin-postfix-log • https://github.com/kazeburo/mackerel-plugin-postfix-log • SMTP • AxlogͰlogrotateΛ௥ै͢Δίʔυॻ͍ͨͷͰ͙͢Ͱ͖ͨ
14. postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile 0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷ஗Ԇ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌ΁ͷ઀ଓʹ͔͔Δ࣌ؒ ૹ৴ͷ஗Ԇ
15. • mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦ਺ͷ૿Ճׂ߹Λ؂ࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅਺ͳͲ • application_log΁ͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠ো֐ʹର͢Δ؂ࢹ
17. • mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔؂ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮૷ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ഑৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ • ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه࿥͠ूܭ
19. • Mackerel Plugin ࡞੒ύλʔϯ • ো֐/ো֐ʹܨ͕Δ໰୊ͷൃݟ • Metrics Plugin ͷ࡞੒ͱαʔό΁ͷ൓ө • ਺೔MetricsΛΈͯͷɺᮢ஋ઃఆɾਵ࣌ݟ௚͠
20. DB ӡ༻ͱ؂ࢹ
21. • ϗετͷεςʔλεΛ؂ࢹ͢Δ • ϗετεςʔλε Standby ͷDB(slave)αʔόΛ production ʹ౤ೖ • ϝϯςφϯε࡞ۀͷͨΊϨϓϦέʔγϣϯ͕ࢭ·͍͕ͬͯͨɺ௨஌͸དྷͳ͍
22. App App consul consul DNS consul consul 1) look up DNS 2) connect DB check mysql uptime before slave slave slave
23. App App consul consul DNS consul consul 1) look up DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave
24. • Consul ͷαʔϏε͕༗ޮʹͳΔͨΊʹ͸ɺmackerelͷεςʔλεͷมߋ͕ ඞཁɻΞϥʔτʹؾ͕ͭ͘Մೳੑˢ • ϗετεςʔλεΛϩʔΧϧͷͲ͔͜ʹॻ͖ࠐΉػೳɺ
 mackerel-agent ʹ͋Δͱศར?
25. • Slave Ͱͷ Trigger ࣮ߦΛकΔ • ProductionͷҰ෦ͷslaveΛɺௐࠪɾूܭ༻ͱͯ͠࢖͑ΔΑ͏ʹ͍ͯ͠Δ • ηϯγςΟϒͳ৘ใʹ͍ͭͯ͸Slave্ͷTriggerͰ࡟আ
26. • MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹ͸SQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ୒͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰ͸جຊ͸Statementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕࢖Θ ΕΔͱɺRow BasedͱͳΔ • ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTrigger͸ɺStatementํࣜͷ৔߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹ͸ҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ
27. • check-mysqlbinlog-rbr-fallbacks • No OSS • mysqlbinlog ίϚϯυ࢖͍ɺRow BasedʹͳͬͨςʔϒϧΛݕग़
28. Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the SQL and fix data and 
 applications if necessary App
29. • ௅ઓ͍ͯ͠ΔݶΓɺ؂ࢹ͸૿͑Δ • Application Server: 175 Metrics • DB Server: 245 Metrics
30. • Ҏ্