メルカリのシステム・サービス監視について/Monitoring Mercari service and servers

メルカリのシステム・サービス監視について
Monitoring seminar in Mercari

1. ϝϧΧϦͷγεςϜɾαʔϏε ؂ࢹʹ͍ͭͯ Monitoring Seminar in Mercari 2017/Good/Meat @kazeburo
2. Me • Masahiro Nagano / ௕໺խ޿ • id:kazeburo • Mercari, Inc
 Principal Engineer
 Site Reliability Engineering (SRE) Team
3. Agenda • Mercariͷ͜Ε·Ͱͱ؂ࢹπʔϧ • MackerelͰͷαʔό؂ࢹ
4. ~ 2014/9 JP ੴङ ΠϯϑϥνʔϜ!5PLZP
5. ~2014/9 • ͘͞ΒΠϯλʔωοτੴङDCͷઐ༻αʔόͱΫϥ΢υΛར༻ • ઐ༻αʔόʹͯZabbixαʔόΛߏங • ʮτϦΨʔʯΛ׆༻ͯ͠͞·͟·ͳ؂ࢹΛߦ͏ • ৚͍݅ࣜͩ͠Ͱෳࡶͳ؂ࢹ͕࣮ݱ • ݱࡏͷ؂ࢹ߲໨ͷϕʔε͕Ͱ͖Δ
6. 2014/9~ US JP ੴङ ΠϯϑϥνʔϜ!5PLZP
7. 2014/9~ • USαʔϏε͕ AWS Oregon region ʹͯ։࢝ • ઐ༻αʔόͱΫϥ΢υͷϚϧνˍϋΠϒϦουߏ੒ • USʹ΋Zabbix ServerΛߏஙͯ͠ɺ౦ژ͔Β؂ࢹ
8. ଟRegion Zabbixͷ՝୊ • Zabbixͷઃఆ͕ͣΕ͍ͯ͘ • ӡ༻͍ͯ͠ΔZabbixͷόʔδϣϯ͕ҟͳΔ • ઐ༻αʔόͱAWSͰएׯҧ͏؂ࢹ߲໨ • JPͰ࡞ΓࠐΜͩ؂ࢹ͕USͰ࠶ݱͰ͖͍ͯͳ͍ • USͰ͚ͩى͖Δ؂ࢹ࿙ΕʹΑΔࣄނ • Zabbix ProxyΛར༻͠ɺ1ݸͷZabbix Server΁ू໿ͳͲͷҊ΋
9. Zabbixͷ՝୊ • Zabbix ࣗମͷӡ༻ • όʔδϣϯΞοϓͷෛ୲ • MySQL ͷෛՙ͕େ͖͘ɺ؂ࢹ஗ԆͳͲ΋ൃੜ • ෳࡶͳτϦΨʔͷ؅ཧ • Ϛ΢εΫϦοΫओମͷઃఆ • όʔδϣϯ؅ཧͳͲΛߦ͍͍ͨ • ؂ࢹͷ௨஌Λվળ͍ͨ͠
10. 2016/1~ US JP ੴङ 43&!5PLZP
11. mackerel ಋೖ • Service Metrics͔Βಋೖ • ؆୯ʹάϥϑ͕ඳ͚ɺ؂ࢹᮢ஋ͷઃఆ͕Ͱ͖Δ • fluentdɺNorikraͱͷ૊Έ߹Θͤ • ZabbixͷτϦΨʔͷҠ২ • τϦΨʔΛPluginͱ࣮ͯ͠૷ • Plugin͸ GitͰ؅ཧ͠ɺAnsibleͰ഑෍
12. mackerel ಋೖ • ؂ࢹπʔϧɺ࣌ܥྻDBͷӡ༻ͷΦϑϩʔυ • Կ΋͠ͳͯ͘΋ຖिόʔδϣϯΞοϓ • JP/USͰͷ؂ࢹ߲໨Λ߹ΘͤΔ • ҟͳΔͱ͜Ζ͸ Ansible templateͳͲͰٵऩ
13. 2017/3~ US JP ੴङ UK 43&!5PLZP
14. 2017/3~ • UK ͰͷαʔϏε։࢝ • UK Λ։࢝͢Δʹ͋ͨͬͯɺ͞Βʹ΋͏ҰͭͷΫϥ΢υΛ࠾༻ • ؂ࢹ͕Ϋϥ΢υԽ͞Ε͓ͯΓɺ৽ͨͳ؂ࢹαʔόͷ௥Ճ͸ඞཁͳ͠ • JP/US ͷ؂ࢹ߲໨͕ͦͷ··ద༻Ͱ͖ɺΠϯϑϥετϥΫνϟͷߏங׬ྃͱͱ ΋ʹ؂ࢹͷઃఆ͕׬ྃ
15. ݱࡏ US JP ੴङ UK 43&&OHJOFFST!+1646, Stackdriver Prometheus
16. ݱࡏ • ϚΠΫϩαʔϏεԽ • GKE ্ͷίϯςφɾαʔϏεͷ؂ࢹͷͨΊʹ Stack DriverɺPrometheusɺ DataDog ͷ׆༻ • αʔόαΠυΤϯδχΞ΋؂ࢹπʔϧΛར༻
17. ͦͷଞͷ؂ࢹ New Relic Kurado άϥϑը૾ΛҰؾʹݟΕΔͷͰศར جຊతͳϝτϦΫε͸ͪ͜ΒͰݟΔ PHPͷ಺෦ͷτϨʔε ΞϓϦέʔγϣϯͷνϡʔχϯάͷࢀߟ
18. MackerelͰͷαʔό؂ࢹ
19. https://speakerdeck.com/kazeburo/mackerel-day
20. ؂ࢹʹ·ͭΘΔ਺ࣈ • ؂ࢹϧʔϧ਺: 278 • Hostຖͷ؂ࢹϧʔϧ਺ • MySQL: 34 • Application: 39 • Search: 37 • Custom Plugin: 50+ (check + metrics + utils)
21. MySQLͷ؂ࢹ߲໨(1/4) • Connectivity • FileSystem % >85% >88% • Swap % >50% >70% • ssh-alive • sshdͷϓϩηε؂ࢹ • global-ip-and-iptables • global ipͷ༗ແͱiptablesͷঢ়ଶ • unbound-resolv • localͷunboudͰ໊લղܾ͕Ͱ͖Δ͔ • unbound-process • unboundͷϓϩηε؂ࢹ • crond-process • crondͷϓϩηε؂ࢹ • uptime • ࠶ىಈ؂ࢹ
22. MySQLͷ؂ࢹ߲໨(2/4) • inode-usage • inode࢖༻཰ >80% >90% • uname-change • unameίϚϯυͷ݁Ռͷdiff؂ࢹ • passwd-change • passwdϑΝΠϧͷdiff؂ࢹ • hostname-changed • hostnameίϚϯυͷ݁Ռͷdiff؂ࢹ • custom.ntpq.synced.remote <0.1 <0.1 • custom.ntpq.offset.seconds >300 >300 (msec) • ntpͷಉظαʔόͱ࣌ࠁͷζϨ • custom.linux-lite.memory.avail <50MB <20MB • ۭ͖ϝϞϦ • custom.linux-lite.cpu-usage.cpu-steal >20% >20% • custom.linux-lite.cpu-usage.cpu-iowait >30% >50% • custom.linux-lite-cpu-usage.cpu-system >8% >8% • ͦΕͧΕͷCPU࢖༻཰(100%্͕ݶ)
23. MySQLͷ؂ࢹ߲໨(3/4) • cutom.linux-lite.loadavg.per-cpu >3 >3 • ίΞ਺ͰׂͬͨϩʔυΞϕϨʔδ • postfix-smtp-alive • SMTPϙʔτͷ֬ೝ • postfix-master-process • postfix masterϓϩηε؂ࢹ • custom.postfix.mailq.queue >100 >5k • postfix mail Ωϡʔ଺ཹ਺ • custom.linux-lite.process.all >2k >2k • custom.linux-lite.process.running >60 >100 • શϓϩηε਺ͱ࣮ߦதͷϓϩηε਺ • mysql-uptime • mysqlͷuptime • custom.mysql-lite.replication-threads.io <0.2 <0.2 • custom.mysql-lite.replication-threads.sql <0.2 <0.2 • ϨϓϦέʔγϣϯͷ֤threadͷঢ়ଶ
24. MySQLͷ؂ࢹ߲໨(4/4) • custom.mysql-lite.replication-behind- master.second >5 >5 • mysqlͷϨϓϦέʔγϣϯ஗Ԇ • custom.mysql-lite.connections.utilization >90 >90 • max_connectionsʹର͢ΔίωΫγϣϯ਺ • custom.mysql-lite.threads.running >1k >2k • mysql্Ͱ࣮ߦதͷεϨου਺ • mysql-slave-sql-error • replicationΤϥʔͷ؂ࢹ • machine-exceptions • αʔόͷϝϞϦΤϥʔ؂ࢹ • raid-disks • αʔόͷRAID/Diskঢ়ଶͷ؂ࢹ
25. ؂ࢹͷҭ͔ͯͨ • ʮ؂ࢹ͸ܧଓతͳςετʯby kazuho • ʮςετϑΝʔετʯͰγεςϜΛߏங͠ͳ͕Β؂ࢹΛ࡞Δ • ϓϩηε؂ࢹ΍ϙʔτ؂ࢹ • ʮϙετϞʔςϜʯͷରԠࡦͱͯ͠؂ࢹΛҭͯΔ • ᮢ஋ͷௐ੔ɺL7Ϩϕϧͷ؂ࢹ
26. end