Observability

Tech Talk given on Observability and it's pillars to multiple groups.

1. Observability ------ Metrics, Monitoring, Alerting Piyush Goel, Feb 2019
2. Disclaimer: Content is not original. Check References for the source.
3. When is a developers job done? 1. After dev-complete? 2. After Staging push? 3. After QA sign-off? 4. After Prod Release?
4. When is a developers job done? 1. After dev-complete? 2. After Staging push? 3. After QA sign-off? 4. After Prod Release? Answer: None of the above!
5. Release to Production is just the beginning!
6. Release to Production is just the beginning! “40% to 90% of the total costs of software are incurred after launch.” • Facts and Fallacies of Software Engineering, Glass R (2002), Addison-Wesley, p-115 • Which Factors affect Software Projects maintenance costs more? Acta Informatica Medica
7. Systems Will Fail!
8. Systems Will Fail … Be Prepared for it!
9. #failure points in a distributed system increase with each new component.
10. What to monitor?
11. What to monitor? Let’s take an example: Platform API’s
12. What to monitor? Let’s take an example: Platform API’s • API Latency (95th percentile, Avg, 99th percentile) • CPU, Load Avg • Memory • Swap • JMX Heap Size • HTTP Error Codes (200, 400, 300) • Exceptions • External API call Latencies • … • And more…….
13. What to monitor? Let’s take an example: Platform API’s • API Latency (95th percentile, Avg, 99th percentile) • CPU, Load Avg • Memory • Swap • JMX Heap Size • HTTP Error Codes (200, 400, 300) • Exceptions • External API call Latencies • And more……. X Servers X Clusters
14. What to monitor? Let’s take an example: Platform API’s • API Latency (95th percentile, Avg, 99th percentile) • CPU, Load Avg • Memory • Swap • JMX Heap Size • HTTP Error Codes (200, 400, 300) • Exceptions • External API call Latencies • And more……. X Servers X Clusters
15. Let’s step back and understand monitoring.
16. Monitoring • Capturing the state of the system to determine its health. ◦ HealthChecks ▪ Is the service running? ▪ Can I do more work? ◦ Metrics ▪ System ▪ Application ▪ Functional • Alerts ◦ Anomalous behaviors - How do you define an anomaly?
17. Monitoring • Alerts - Known Failures ◦ Knowledge Based ◦ Reactive (post-outages) What about the unknown failures?
18. Observability
19. Observability https://medium.com/@copyconstruct/monitoring-and-observability-8417d1952e1c
20. Observability The bygone era!!
21. Observability • https://www.vividcortex.com/blog/monitoring-isnt-observability
22. Observability • https://theagileadmin.com/2018/02/16/monitoring-and-observability/ Observability is a measure of how well the internal states of a system can be inferred from knowledge of its external outputs.
23. Observability - Internal States • Context Specific ◦ Web Servers ▪ Availability ▪ Incoming Request Rate ▪ Latency ▪ HTTP Failures ◦ Micro-Services ▪ Success Rate ▪ Functionalities ◦ Message Queue ▪ Queue Length ▪ Consumer/Producer Count Needs Instrumentation! While writing code
24. Observability - External Outputs • Health Checks • Metrics • Logging • Tracing
25. Observability - Health Checks • HealthChecks ◦ Is the service running? ◦ Can I do more work? • Methods ◦ Broadcast - Gossip Protocols (Cassandra) ◦ Register - Service Discovery ◦ Health endpoints - ELB, HAProxy, Nginx
26. Observability - Metrics • External State at a broad scope (Time dimension) ◦ System ◦ Application ▪ Success Rate/Failure Rate ▪ Latency (internal/external) ▪ Error Codes ▪ Exceptions ◦ Business/Functional ◦ Order Rate (Regular, Cancel, Return) ◦ Payments ◦ Conversion Rates ◦ Coupon Issual/Redemption ◦ Points Issued/Redeemed
27. Observability - Metrics • Meaningful Metrics** - Generous • Alerts - Judicious • Low Cardinality ◦ Keep a Watch! ◦ Don’t emit for users/orders. We use Logs for that! • Provide system summary • Questions: ▪ How many transactions failed? ▪ How many logins succeeded? ** https://queue.acm.org/detail.cfm?id=3309571 - Must Read
28. Observability - Metrics • Tools ◦ Graphite ◦ InfluxDB ◦ Prometheus ◦ OpenTSDB ◦ Scuba (Facebook) ◦ Apache Druid
29. Observability - Logging • Understanding at a smaller scope ◦ Request, customer, transaction • Ask Questions: ◦ Why couldn’t the customer place an order? ◦ Why did the transaction fail? • Centralised - ElasticSearch, Splunk • Searchable - Indexed • Correlatable - Common Key (Request Id)
30. Observability - Logging
31. Observability - Logging (Anatomy)
32. Observability - Logging • Tools ◦ Splunk ◦ ELK ◦ SumoLogic ◦ RLA
33. Observability - Tracing • Dissect a request into sub-paths. (Spans) • Profile system usage at a span level. • Extract Insights Tools: • Google Dapper (https://ai.google/research/pubs/pub36356) • Twitter Zipkin (https://zipkin.io/) • Open Jaeger (https://www.jaegertracing.io/) • New Relic
34. Observability Spectrum
35. We have Metrics. What to alert on?
36. Service Level Objectives (SLO)
37. Service Level Objectives (SLO) * https://landing.google.com/sre/sre-book/chapters/service-level-objectives/ * https://www.youtube.com/watch?v=tEylFyxbDLE • Defines a Quantifiable Goal for a service. • Measure the goal - Represents the User Experience/Delight Factor. • First step before writing a new service. Work backwards • Have as few SLO’s as possible. ◦ Represents the system behaviour.
38. Service Level Objectives (SLO) - Exercise • Cart Service • Payments Service • Card Generation • Order Management Service • Communication Engine
39. Happy Observing!
40. References • Debugging Production Systems : https://www.youtube.com/watch?v=YlrAakN90D0 • Pierre Vincent - How to build observable Distributed systems? https://www.youtube.com/watch?v=ACL_YVPD3gw • Charity Majors - Observability for Emerging Infra: What Got You Here Won't Get You There" https://www.youtube.com/watch?v=1wjovFSCGhE • Caitee McAfree - Of the Order of Billions: Building Observability at Twitter https://www.youtube.com/watch?v=SC6XuD1tgcQ • https://eng.uber.com/observability-at-scale/