Log all the things!

Honza Král's talk from Europython 2016:

Centralized logging (and the Elastic stack) is proving itself to be a very useful tool in managing a production infrastructure. When combined with other data sources (application logging, business data, …) it can provide even more insight.

This talk is an introduction into the area with some overview of the motivation, tools and techniques that can prove useful. We will show how the open source ELK (Elasticsearch Logstash and Kibana) stack can be used to implement this.

It is geared towards people familiar with the DevOps concept that are looking to improve their lives by introducing smarter tools.

Transcript

1. Log all the things! Honza Král @honzakral
2. Logs?
3. Log lines Twitter feed Invoices Metrics Events!
4. Why?
5. What happened last Tuesday?
6. Multiple machines Multiple logs Analysis/Discovery Time period Grep?
7. Time? Time?! Time! apache unix timestamp log4j postfix.log ISO 8601 [23/Jan/2014:17:11:55 +0000] 1390994740 2009-01-01T12:00:00+01:00 [2014-01-29 12:28:25,470] Feb 3 20:37:35
8. Web Server logs VS Load Balancer see immediately that caching is off static files leaking to gunicorn Web Server VS Database 500s VS Deploys new version has a bug Traffic VS Ad Campaigns Correlate events
9. Central storage Even for data from different systems Enriched data IP -> location, hostname URL -> author, product, category Search user:honza status:404 Analysis Visualisations for easy pattern discovery Ideal state
10. Centralised Logging
11. Steps Collect data Parse data Enrich data Store data Search and aggregate Visualize data
12. Elastic Stack
13. Steps in Elastic Stack Collect data Parse data Enrich data Store data Search and aggregate Visualize data
14. Steps in Elastic Stack Collect data Parse data Enrich data Store data Search and aggregate Visualize data
16. metricbeat: modules: - module: redis metricsets: ["info"] hosts: ["host1"] period: 1s enabled: true - module: apache metricsets: ["info"] hosts: ["host1"] period: 30s enabled: true filebeat: prospectors: - paths: - "logs/access.log" document_type: access multiline: pattern: ^# negate: true match: after protocols: http: ports: [80, 8000] mysql: ports: [3306] redis: ports: [6379] pgsql: ports: [5432] thrift: ports: [9090] output: logstash: hosts: ["localhost:5044"]
18. Inputs Monitoring collectd, graphite, ganglia, snmptrap, zenoss Datastores elasticsearch, redis, sqlite, s3 Queues kafka, rabbitmq, zeromq Logging beats, eventlog, gelf, log4j, relp, syslog, varnish log Platforms drupal_dblog, gemfire, heroku, sqs, s3, twitter Local exec, generator, file, stdin, pipe, unix Protocol imap, irc, stomp, tcp, udp, websocket, wmi, xmpp
19. Filters aggregate alter anonymize collate csv cidr clone cipher checksum date dns drop elasticsearch extractnumbers environment elapsed fingerprint geoip grok i18n json json_encode kv mutate metrics multiline metaevent prune punct ruby range syslog_pri sleep split throttle translate uuid urldecode useragent xml zeromq ...
20. Outputs Store elasticsearch, gemfire, mongodb, redis, riak, rabbitmq, solr Monitoring ganglia, graphite, graphtastic, nagios, opentsdb, statsd, zabbix Notification email, hipchat, irc, pagerduty, sns Protocol gelf, http, lumberjack, metriccatcher, stomp, tcp, udp, websocket, xmpp External service google big query, google cloud storage, jira, loggly, riemann, s3, sqs, syslog, datadog External monitoring boundary, circonus, cloudwatch, librato Local csv, dots, exec, file, pipe, stdout, null
22. Open Source
 
 Document-based
 
 Based on Lucene 
 JSON over HTTP Distributed Search Engine
23. Cluster Collection of Nodes Index Collection of Shards Shard Unit of scale Distributed across cluster Primary and replica Data Management node 1 orders products 2 1 4 1 node 2 orders products 2 2 node 3 orders 3 4 1 3 products
24. Time based data flow Current replicas to speed up search on stronger boxes Week old snapshot keep only 1 replica Month old move to weaker boxes 2 months close the indices 3 months delete
29. Architecture Enrich Visualize Collect Store
30. Logging and Python
31. Track metrics execution time query time # of queries Include metadata user_id content Log as JSON Enhance your logs
32. Add structured info Track info through services Log to file Add filebeat to read the file Structlog
33. Thanks! Honza Král @honzakral