Share
Embed
Transcript
30 slides
-
Me • Masahiro Nagano • @kazeburo • Mackerel Ξϯόαμʔ •
Ұ൪͖ͳ mackerelϓϥάΠϯ mackerel-plugin-axslog (ࣗ࡞)ͩͳɻ ʹཱͭͱ͜ΖͰ diff-detector(ࣗ࡞) -
0 2 4 6 8 accesslog(ltsv) axslog(ltsv) axslog(json) 1.17s 0.584s
7.976s 120ສߦ(20000req/sec)ͷϩάΛॲཧ͢Δ࣌ؒ -
[plugin.metrics.axslog-web-googlebot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=googlebot \ --logfile /var/log/nginx/access.log --filter Googlebot"
[plugin.metrics.axslog-web-adsbot] command = "/usr/local/bin/mackerel-plugin-axslog --key-prefix=adsbot \ --logfile /var/log/nginx/access.log --filter AdsBot-Google" -
postfixlog.total_delay.average 0.240476 1555681849 postfixlog.total_delay.99_percentile 0.250000 1555681849 postfixlog.total_delay.95_percentile 0.240000 1555681849 postfixlog.total_delay.90_percentile
0.240000 1555681849 postfixlog.recving_delay.average 0.040000 1555681849 postfixlog.recving_delay.99_percentile 0.040000 1555681849 postfixlog.recving_delay.95_percentile 0.040000 1555681849 postfixlog.recving_delay.90_percentile 0.040000 1555681849 postfixlog.queuing_delay.average 0.000476 1555681849 postfixlog.queuing_delay.99_percentile 0.010000 1555681849 postfixlog.queuing_delay.95_percentile 0.000000 1555681849 postfixlog.queuing_delay.90_percentile 0.000000 1555681849 postfixlog.connection_delay.average 0.090000 1555681849 postfixlog.connection_delay.99_percentile 0.090000 1555681849 postfixlog.connection_delay.95_percentile 0.090000 1555681849 postfixlog.connection_delay.90_percentile 0.090000 1555681849 postfixlog.transmission_delay.average 0.090000 1555681849 postfixlog.transmission_delay.99_percentile 0.090000 1555681849 postfixlog.transmission_delay.95_percentile 0.090000 1555681849 postfixlog.transmission_delay.90_percentile 0.090000 1555681849 postfixlog.transfer_num.2xx_count 1.615385 1555681849 postfixlog.transfer_num.4xx_count 0.000000 1555681849 postfixlog.transfer_num.5xx_count 0.000000 1555681849 postfixlog.transfer_total.count 1.615385 1555681849 postfixlog.transfer_ratio.2xx_percentage 100.000000 1555681849 postfixlog.transfer_ratio.4xx_percentage 0.000000 1555681849 postfixlog.transfer_ratio.5xx_percentage 0.000000 1555681849 ड৴࣌ͷԆ Ωϡʔʹೖ͍ͬͯΔ࣌ؒ ૹ৴ઌͷଓʹ͔͔Δ࣌ؒ ૹ৴ͷԆ -
• mackerel-plugin-log-incr-rate • https://github.com/kazeburo/mackerel-plugin-log-incr-rate • ͋Δϩάʹର͢Δผͷϩάͷߦͷ૿Ճׂ߹Λࢹ͢Δ • accesslogʹର͢Δapplication_log, errorlogͷ݅ͳͲ •
application_logͷॻ͖ࠐΈ͕രൃͯ͠ϩάܭଌʹӨڹͨ͠োʹର͢Δࢹ -
• mackerel-plugin-gzip-response • Not OSS • ϨεϙϯεͷGzipѹॖ͕ߦΘΕ͍ͯΔ͔ࢹ • Ұ෦ΫϥΠΞϯτͷ࣮ͰGzip͕ແޮʹͳͬͨ··ɺେ͖ͳ৴ͱͳΔϚε λʔߋ৽͕ߦΘΕɺωοτϫʔΫʹෛՙ͕͔͔Δࣄଶ͕͋ͬͨ
• ϩάʹϨεϙϯεͷ Conent-Encoding ϔομΛه͠ूܭ -
• Mackerel Plugin ࡞ύλʔϯ • ো/োʹܨ͕Δͷൃݟ • Metrics Plugin ͷ࡞ͱαʔόͷө
• MetricsΛΈͯͷɺᮢઃఆɾਵ࣌ݟ͠ -
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB check mysql uptime before slave slave slave -
App App consul consul DNS consul consul 1) look up
DNS 2) connect DB A) check mysql uptime After Mackerel batch batch batch B) check host status file B’ ) update host status file by cron slave slave slave -
• MySQLͱϨϓϦέʔγϣϯͱTrigger • MySQLͷϨϓϦέʔγϣϯͷํࣜʹSQL͕సૹ͞Εͯ͘ΔStatementํࣜͱɺ σʔλ͕ૹΒΕͯ͘ΔRow Basedํࣜɺ·ͨͲͪΒ͔͕ࣗಈͰબ͞ΕΔMixed ํ͕ࣜ͋ΔɻMixedํࣜͰجຊStatementํ͕ࣜͩɺඇܾఆੑΫΤϦ͕Θ ΕΔͱɺRow BasedͱͳΔ •
ϨϓϦέʔγϣϯΛड͚Δslaveαʔό্ͰͷTriggerɺStatementํࣜͷ߹ͷ Έ࣮ߦ͞ΕΔ • SlaveͰtriggerΛ҆શʹར༻͢ΔͨΊʹҙਤ͠ͳ͍ Row basedͳϨϓϦέʔ γϣϯΛൃݟ͢Δඞཁ͕͋Δ -
Master Backup Anon Slave Trigger binlog binlog check-mysqlbinlog-rbr-fallbacks Detect the
SQL and fix data and applications if necessary App