ログのメトリックを取ってみる話

logback / Prometheus

Transcript

1. ϩάͷϝτϦοΫΛऔͬͯΈΔ࿩ @atty303 / AI Studio Dev Group © 2016 CyberAgent, Inc. 1
2. ELK stack ͱ͔Ͱϩάऩू͢ Δ࿩͸Α͋͘Δ © 2016 CyberAgent, Inc. 2
3. ϩάͷϝτϦοΫΛܭଌ͢ Δ࿩͸͋Μ·Γͳͦ͞͏? © 2016 CyberAgent, Inc. 3
4. ͜Μͳײ͡Ͱܭଌͯ͠·͢ 4 DEBUG Ϩϕϧ͔͠ग़͍ͯͳ͍ͷͰ͍·͍ͪ໘നΈ͸ ͳ͍… © 2016 CyberAgent, Inc. 4
5. 4 ϞχλϦϯάʹ Prometheus Λར༻ 4 Prometheus ެࣜͷ Java ΫϥΠΞϯτ 4 https://github.com/prometheus/ client_java 4 simpleclient_logback Ϟδϡʔϧ 4 Ϩϕϧผʹϝοηʔδ਺Λܭଌͯ͠ϝτϦ οΫͱͯ͠ग़ྗͯ͘͠ΕΔ Logback ͷ Appender © 2016 CyberAgent, Inc. 5
6. logback.xml <configuration> <appender name="COUNTER" class="io.prometheus.client.logback.InstrumentedAppender"> </appender> <appender name="STDOUT" class="ch.qos.logback.core.ConsoleAppender"> <filter class="ch.qos.logback.classic.filter.ThresholdFilter"> <level>${LOG_STDOUT_LEVEL:-INFO}</level> </filter> <encoder> <pattern>%.-1level %d{MMdd HH:mm:ss.SSS} %thread [%logger{36}] %msg%n</pattern> </encoder> </appender> <root level="DEBUG"> <appender-ref ref="COUNTER" /> <appender-ref ref="STDOUT" /> </root> </configuration> © 2016 CyberAgent, Inc. 6
7. ϙΠϯτ 4 ϧʔτϩΨʔ͸શϩάΛ appender ʹૹΔ 4 STDOUT appender ͷ ThreasholdFilter Ͱ ৑௕ͳϩάΛϑΟϧλϦϯά 4 COUNTER appender ͕ϩάϨϕϧผͷϝο ηʔδ਺ΛΧ΢ϯτ © 2016 CyberAgent, Inc. 7
8. ϝτϦοΫܭଌͯ͘͠ΕΔ $ curl http://somewhere:58165/metrics # HELP logback_appender_total Log4j log statements at various log levels # TYPE logback_appender_total counter logback_appender_total{level="debug",} 8.9142634E7 logback_appender_total{level="warn",} 0.0 logback_appender_total{level="trace",} 0.0 logback_appender_total{level="error",} 2728.0 logback_appender_total{level="info",} 11.0 © 2016 CyberAgent, Inc. 8
9. ϩάϨϕϧͷఆٛ DEBUG: ։ൃऀʹ͔͠ඞཁͳ͍৘ใ INFO: ΞϓϦέʔγϣϯͷॲཧঢ়گͳͲਖ਼ৗಈ࡞ Λ͍ͯ͠Δ࣌ͷ৘ใ WARN: Τϥʔ͕ग़͕ͨճ෮ग़དྷͨ࣌ͷ৘ใ ERROR: Τϥʔ͕ग़ͯճ෮ग़དྷ͖ͳ͍࣌ͷ৘ใ 4 ͪΌΜͱϩάϨϕϧͷҙຯ෇͚Λ͍ͯ͠ͳ͍ͱ ܭଌͯ͠΋ແҙຯ… © 2016 CyberAgent, Inc. 9
10. Ξϥʔτϧʔϧ IF rate(logback_appender_total{job="axion-inserter-admin",level="error"}[1m]) > 1 FOR 1m LABELS { severity = "LOW" } ANNOTATIONS { summary = "Inserter: {{ $value }} error messages found at {{ $labels.instance }}" } 4 ͱΓ͋͑ͣ ERROR ϩά͕ग़͍ͯͨΒΞϥʔ τ 4 ϝοηʔδ͸ίϯςφΛݟʹߦ͘… nomad fs <alloc-id> alloc/logs/inserter.stdout.0 © 2016 CyberAgent, Inc. 10
11. …ͯͳӡ༻Λͯ͠·͢ © 2016 CyberAgent, Inc. 11
12. ྑ͍ͱ͜Ζ 4 ϩάऩूΑΓѹ౗తʹτϥϑΟοΫ͕খ͍͞ͷ ͰؾָʹಋೖͰ͖Δ 4 ϩάϨϕϧ͚ͩͰ΋ҙ֎ͱগͳ͍࿑ྗͰ؂ࢹͰ ͖Δ 4 ؂ࢹΛҙࣝͨ͠ϩάग़ྗ͸ඞཁ © 2016 CyberAgent, Inc. 12
13. ѱ͍ͱ͜Ζ 4 ໰୊͕ى͖ͨͱ͖͸݁ہϝοηʔδΛݟʹߦ͔ ͳ͍ͱ͍͚ͳ͍ 4 ϩάऩूͷ࢓૊Έ͕༻ҙ͞Ε͍ͯΕ͹ͦͷ΄͏ ͕ྑ͍ © 2016 CyberAgent, Inc. 13
14. GCP ͷ Stackdriver ͸͏Β΍·͍͠ © 2016 CyberAgent, Inc. 14
15. ׬ © 2016 CyberAgent, Inc. 15