Visibility for web developers

Transcript

1. Visibility for Web Developers Bruno Renié Django Island 2014
2. Working with ops & support staff Large infrastructure, lots of moving parts whoami: lead dev @ exoscale
3. People need to know what's going on, anytime Not everyone is a developer SSH, grep and tail not acceptable
4. Goal: visibility on infrastructure & apps
5. Errors Events Metrics
6. Errors
7. Easy: use sentry
8. Great tool for developers, but… People also need to know about things that are not broken
9. Events
10. Event == log call Timestamped information about something worth knowing
11. Easy to emit logs Searching on remote machines, not so much
12. Centralized logging Agent forwards logs to an aggregator Aggregator stores logs and provides a search interface
13. Aggregator Hosts logstash+elasticsearch kibana log shippers
14. Collecting logs rsyslog / syslog-ng logstash-forwarder (lumberjack) logstash
15. In Python logging.handlers.SyslogHandler
16. Centralizing isn't enough
17. A common way to log things import logging logger = logging.getLogger(__name__) logger.info("user {} logged in".format(user.email))
18. Not much information No structure
19. Structured logging Record = set of key-value pairs event="user logged in" user="test@example.com" ip="1.2.3.4"
20. In Python: pip install structlog Hynek Schlawack logger = structlog.get_logger() logger.info("user logged in", user="test@example.com" ip="1.2.3.4") {"event": "user.login", "user": "test@example.com", "host": "1.2.3.4"}
21. Contextual loggers log = logger.bind(user=request.user.email) log.info("payment", amount=120, currency="CHF") {"event": "payment", "user": "test@example.com", "amount": 120, "currency": "CHF"}
22. structlog JSON elasticsearch Rich data for building dashboards
23. Demo
24. Marketing: "Who signed up today?" Finance: "How much income in the past week?" Support: "What did this user do in the past 24 hours?"
25. Events Centralize logs Add structure Profit logstash, elasticsearch structlog kibana
26. Metrics
27. Time series data Continuous, regular intervals Big player: graphite ecosystem carbon whisper graphite
28. <identifier> <value> <timestamp> db1.load.midterm 4.6 1399842778 Measure at regular intervals, send using the carbon line protocol echo " … " | nc host port
29. Traditional metrics stack agents collectd, apps, scripts carbon whisper graphite ingests metrics stores metrics renders graphs
30. Counting and timing: statsd In-memory buffer Aggregates and flushes to carbon at regular intervals
31. In Python pystatsd django-statsd-mozilla from stastd import StastClient statsd = StatsClient() @statsd.timer("something") def expensive_computation(): # …
32. Graphing Graphite: rendering API (json / PNG / etc) Countless dashboard apps Grafana Graph-explorer etc
33. Demo
34. What about alerts? Riemann: metrics hub/proxy App server overloaded, full disk, etc Analyzes metrics computes aggregates notifies
35. Demo
36. we are hiring! Questions! @brutasse Image credits: http://en.wikipedia.org/wiki/Explosion
37. Resources Events Metrics structlog.org rsyslog logstash elasticsearch kibana carbon / whisper graphite-api grafana collectd statsd riemann.io bernhard
38. Hosted services (not exhaustive) Logging Metrics Loggly Papertrail Hosted Graphite Librato JSON / structured logging support?