Distributed Metrics and Log Aggregation - JavaLand

This presentation was given at JavaLand 2015. Find more information at https://www.innoq.com/de/talks/2015/03/distributed-log-aggregation-javaland-2015/

Transcript

1. Distributed Metrics and Log Aggregation Alexander Heusingfeld & Tammo van Lessen
4. ...so you start to disassemble your monoliths...
5. ...and then you think about the infrastructure
6. Architectural Decisions Domain architecture Macro architecture Micro architecture Ą Ą Ą
7. Scenario: Big Shop
8. That wouldn't have happened with proper logging!
9. What makes good logging? What identifies a good log message? Which log level should I use when? Should I log into files? What format? Ą Ą Ą
10. Some recommendations Log messages should have a uniform style. Log violations of assumptions. Use markers to make log streams filterable. Prefer machine-readable log formats over human-readable. Identify correlation tokens and attach them to the log event. Collect and store logs in a central repository. Ą Ą Ą Ą Ą Ą
11. Default Levels Files? Warn only. Logstash & Co? Info. Magic bugs + advanced setup? Debug, or even trace.
12. Async Appenders (LMAX, MemoryMappedFileAppender) Routing Properties Reconfiguration (Auto load, JMX,...) Audit logs Markers / Log levels ... Ą Ą Ą Ą Ą Ą Ą
13. Thread Context ThreadContext.put("loginId", login); logger.error("Something bad happened!"); ThreadContext.clear(); + Layout: %-5p: [%X{loginId}] %m%n Log: ERROR: [John Doe] Something bad happened!
14. Thread Context (2) ThreadContext.put("loginId", login); logger.error("Something bad happened!"); ThreadContext.clear(); + JSON Layout: Log: { "@version" => "1", "@timestamp" => "2014-04-29T14:21:14.988-07:00", "logger" => "com.example.LogStashExampleTest", "level" => "ERROR", "thread" => "Test worker", "message" => "Something bad happened!", "Properties" => { "loginId" => "John Doe" } }
15. Requirements in a distributed environment Aggregate logs in different formats from different systems. Search & Correlate Visualize Alert on complex correlations. Ą Ą Ą Ą
16. Tools of Trade
17. Logstash Architecture
18. Logstash – Hands on!
19. A Logstash Cluster From the Logstash docs
20. … and there are others, too! Apache Flume (ASL 2.0) FluentD (ASL 2.0) Graylog 2 (GPL) Loggly (commerical) Splunk (commerical)
21. Logging is cool. And I can use it to collect metrics as well, right?
22. Yes, you can! But you shouldn't!
23. Metrics Business Metrics Application Metrics System Metrics Ą Ą Ą
24. Continuous Delivery & Metrics?
25. Continuous Delivery & Metrics?
26. Gauges An instrument that measures a value.
27. Counters A counter is a simple incrementing and decrementing integer.
28. Meters A meter measures the rate at which a set of events occur.
29. Histograms A Histogram measures the distribution of values.
30. Timers
40. Dashboards
41. Graphite
42. Cubism.js Credits: Michael Bostock Mirror Offset 1 − +
44. Comparisons var cube = context.cube("http://..."), primary = cube.metric("sum(request)"), secondary = primary.shift(-7 * 24 * 60 * 60 * 1000);
45. ... Dashing
46. Best practices Measure everything! Counters ./. Meters Metrics are cheap, but not for free. Retention Policies Get rid of silos Correlate your data ...to make better decisions Ą Ą Ą Ą Ą Ą Ą
47. Prevent the apocalypse! Logging shows events. Metrics shows state. Don't fly blind!
48. Thanks for your attention! Alexander Heusingfeld | «@goldstift Tammo van Lessen | «@taval https://www.innoq.com/
49. Credits Ą Buuz and Woody Ą Monolith by Ron Cogswell Ą Dave - Wrapping up monolith tins Ą Pleuntje - connected Ą CPU by mbostock Ą Mess by Rev Stan Ą Pay Here by Marc Falardeau Ą Cockpit by Ronnie Rams Ą Stream by Phil Whitehouse Ą Magnifier by John Lodder (Flickr) Ą Flying Saucer, Cup, and Teapot! by Mr Thinktank Ą Ice berg by Derek Keats Ą Gas Meters by mxmstryo (Flickr) Ą Gauge Stock by Andrew Taylor (Flickr) Ą Counter by Marcin Wichary (Flickr) Ą Histogram of legos by color frequency by Jeff Boulter (Flickr) Ą pomodoro timers by Paul Downey (Flickr) Ą Zombie Apocalypse by pasukaru76