LinuxFest Northwest 2016 - Monitoring 101

You need to monitor only a few machines and applications before identifying and fixing issues in your environment becomes very complicated. Throw in the type of dynamic infrastructure provided by cloud providers and container orchestration, and your static monitoring strategies will most likely not scale. Knowing which metrics to watch and how to troubleshoot based on those metrics will help you solve problems more quickly. In this session, we will look at a framework for your metrics and how to use it to find solutions to the issues that come up. We will cover the three types of monitoring data; what to collect; what should trigger an alert (avoiding an alert storm and pager fatigue); and how to follow the resources to find the root causes of problems. This focus of this session is not tool specific, so attendees will leave with strategies and frameworks they can implement in environments today regardless of the platforms and tools they use.

Transcript

1. LinuxFest Northwest 2016
2. Monitoring 101 LinuxFest Northwest April 24, 2016 Ilan Rabinovitch Director, Technical Community
 Datadog
3. $ finger ilan@datadog [datadoghq.com] Name: Ilan Rabinovitch Role: Director, Technical Community 
 Interests: * Open Source * Large scale web operations * Monitoring and Metrics * Planning FL/OSS Community Events
4. • SaaS based infrastructure and app monitoring • Open Source Agent • Time series data (metrics and events) • Processing nearly a trillion data points per day • Intelligent Alerting • We’re hiring! (www.datadoghq.com/careers/) Datadog Overview
5. Operating Systems, Cloud Providers (AWS), Containers, Web Servers, Datastores, Caches, Queues and more... Monitor Everything
7. $ cat ~/.plan 1. Intro and Background: What is DevOps? 2. The Challenge: Monitoring Dynamic Infrastructure 3. Finding the Signal: How do we know what to monitor? 4. Implementation
8. Our Focus Area Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA
9. Culture “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations” - Melvin E. Conway
13. Follow @honest_update on Twitter
16. Sharing Looping Back on Culture Describe the problem as your “enemy” not each other Learn Together
17. Sharing Using and Sharing the same metrics and measurements across teams is key to avoiding misunderstandings.
18. Sharing Using and Sharing the same metrics and measurements across teams is key to avoiding misunderstandings.
19. Our Focus Area Culture Automation Metrics Sharing Damon Edwards and John Willis DevOps Day LA
20. Collecting data is cheap;
 not having it when you need it can be expensive
21. Instrument all the things!
23. You’re in the cloud and it's everything you dreamed of! Autoscaling Infinite Storage Managed 
 Databases Container Orchestration Private Clouds
24. More info at: www.datadoghq.com/docker-adoption/
27. Source: http://bit.ly/1SvvbuP
28. Source: http://bit.ly/1RQRsXW
29. Operational Complexity Increases with.. • Number of things to measure
 • Velocity of change
30. How much we measure? 1 instance • 10 metrics from CloudWatch 1 operating system (e.g., Linux) • 100 metrics 50~ metrics per application
32. How much we measure? 1 instance • 10 metrics from CloudWatch 1 operating system (e.g., Linux) • 100 metrics 50~ metrics per application
 N containers • 150*N metrics
33. Operational Complexity 100 instances 400 containers
34. Operational Complexity: Scale 160 metrics per host 640 metrics per host Assuming 4 containers per host
35. Operational Complexity: Scale 100 instances 64,000 metrics Assuming 4 containers per host
36. How much we measure? 1 instance • 10 metrics from CloudWatch 1 operating system (e.g., Linux) • 100 metrics 50~ metrics per application
 N containers • 150*N metrics Metrics Overload!
37. Operational Complexity Increases with.. • Number of things to measure
 • Velocity of change
38. Source: Datadog
39. Source: http://bit.ly/1qFylWK
40. Operational Complexity Increases with.. • Number of things to measure
 • Velocity of change
43. More Details at: http://www.datadoghq.com/blog/monitoring-101-alerting/
44. Finding Signal - Categorizing Your Metrics
49. Examples: NGINX - Metrics Work Metrics: 
 • Requests Per Second Dropped Connections • Request Time • Error Rates (4xx or 5xx) • Success (2xx) Resource Metrics:
 • Disk I/O • Memory • CPU • Queue Length
50. Examples: NGINX - Events • Configuration Change • Code Deployment • Service Started / Stopped • etc
51. Examples: Events
52. When to let a sleeping engineer lie?
53. When to alert?
55. Recurse until you find root cause
56. How does your current monitoring 

58. Trending vs Alerting Many Point Solutions How do they all fit together? Too Many Tools
60. Pick tools that let you aggregate many data sources Too Many Tools
61. Cryptic Alerts
62. Cryptic Alerts W H A T ?
64. Informative and Actionable Alerts Why is this important? What do I do about it? Who do I call next if I get stuck? EVERY ALERT MUST BE ACTIONABLE
65. Averages Are Lies You can’t provision for your average traffic. Keep the real data.
66. Static configurations tracking dynamic infrastructure Static vs Dynamic
67. Host Centric
68. Service Centric
69. Tags All the Way Down
72. Asking Better Questions “Monitor all containers running image web in region us-west-2 across all availability zones that use more than 1.5x the average memory on c3.xlarge”
73. Asking Better Questions “90% of all web requests are taking more than 0.5s to process and respond.”
74. Custom Metrics • Instrument custom applications
 • You know your key transactions best.
 • Use async protocols like Etys’ STATSD
75. Source: http://bit.ly/1NoW6aj
76. Resources Monitoring 101: Alerting 
 https://www.datadoghq.com/blog/monitoring-101-alerting/ Monitoring 101: Collecting the Right Data https://www.datadoghq.com/blog/monitoring-101-collecting-data/ Monitoring 101: Investigating performance issues https://www.datadoghq.com/blog/monitoring-101-investigation/
 The Power of Tagged Metrics https://www.datadoghq.com/blog/the-power-of-tagged-metrics/ Monitoring Sucks Project https://github.com/monitoringsucks/