Infrastructure & System Monitoring using Prometheus.pdf

Hosted during DevoxxUK (2017)

https://github.com/mpas/infrastructure-and-system-monitoring-using-prometheus

2. Some stuff about me... • Mostly doing cloud related stuff ◦ Java, Groovy, Scala, Spring Boot, IOT, AWS, Terraform, Infrastructure • Enjoying the good things • Chef leuke dingen doen == “trying out cool and new stuff” • Currently involved in a big IOT project • Movie & Netflix addict
3. Agenda • Monitoring ◦ Introducing you to a Scary Movie • Prometheus overview (demo’s) ◦ Running Prometheus ◦ Gathering host metrics ◦ Introducing Grafana ◦ Monitoring Docker containers ◦ Alerting ◦ Instrumenting your own code ◦ Service Discovery (Consul) integration
4. ..Quick Inventory..
5. I am going to introduce you to some bad movies
6. Bad, very bad, extremely bad movies
12. Commonality between these movies?
13. Watching a bad movie in our case is exactly what we want!
14. Monitoring
16. Our scary movie “The Happy Developer” • Lets push out features • I can demo so it works :) • It works with 1 user, so it will work with multiple • Don’t worry about performance we will just scale using multiple machines/processes • Logging is into place
17. But then the scary stuff begins
18. Did anyone notice? Disaster Strikes
19. Logging “recording to diagnose a system” Monitoring “observation, checking and recording” Logging != Monitoring
20. Vital Signs
21. Why Monitoring? • Know when things go wrong ◦ Detection & Alerting • Be able to debug and gain insight • Detect changes over time and drive technical/business decisions • Feed into other systems/processes (e.g. security, automation)
22. What to monitor? IT Network Operating System Services Applications Capture Monitoring Information Functional Monitoring Operational Monitoring metric data
23. Functional Monitoring Applications Services event data
24. Houston we have Storage problem! Storage metric data metric data metric data metric data metric data metric data metric data metric data metric data How to store the mass amount of metrics and also making them easy to query?
25. Time Series - Database • Time series data is a sequence of data points collected at regular intervals over a period of time. (metrics) ◦ Examples: ▪ Device data ▪ Weather data ▪ Stock prices ▪ Tide measurements ▪ Solar flare tracking • The data requires aggregation and analysis Time Series Database metric data • High write performance • Data compaction • Fast, easy range queries
26. Time Series - Data format
27. Source: http://db-engines.com/en/ranking/time+series+dbms http://db-engines.com/en/ranking/time+series+dbms
28. Prometheus Overview
29. Prometheus Prometheus is an open-source systems monitoring and alerting toolkit originally built at SoundCloud. It is now a standalone open source project and maintained independently of any company. https://prometheus.io Implemented using
30. Prometheus Components • The main Prometheus server which scrapes and stores time series data • Client libraries for instrumenting application code • A push gateway for supporting short-lived jobs • Special-purpose exporters (for HAProxy, StatsD, Graphite, etc.) • An alertmanager • Various support tools • WhiteBox Monitoring instead of probing [aka BlackBox Monitoring]
31. Prometheus Overview
32. List of Job Exporters • Prometheus managed: ◦ JMX ◦ Node ◦ Graphite ◦ Blackbox ◦ SNMP ◦ HAProxy ◦ Consul ◦ Memcached ◦ AWS Cloudwatch ◦ InfluxDB ◦ StatsD ◦ ... • Custom ones: ◦ Database ◦ Hardware related ◦ Messaging systems ◦ Storage ◦ HTTP ◦ APIs ◦ Logging ◦ … https://prometheus.io/docs/instrumenting/exporters/
33. Demo Structure
34. Demo: Run Prometheus (native)
36. Code Demo “Running Prometheus Native”
37. Demo: Run Prometheus using Docker
38. 38 → → → →
39. Code Demo “Running Prometheus Dockerized”
40. Demo: Add host metrics
41. → → → → → → →
43. Code Demo “Add host metrics”
44. Demo: Grafana 44
45. → → You get the idea :)
46. Code Demo “Grafana”
47. Demo: Monitor Docker containers
48. Code Demo “cAdvisor”
49. Demo: Alerting
50. Alerting Configuration • Alert Rules ◦ What are the settings where we need to alert upon? • Alert Manager ◦ Where do we need to send the alert to?
51. → →
52. → → → → → →
53. → →
55. Code Demo “Alerting -> The Alert Manager”
56. Instrumenting your own code
57. Instrumenting your own code! • Counter ◦ A cumulative metric that represents a single numerical value that only ever goes up • Gauge ◦ Single numerical value that can arbitrarily go up and down • Histogram ◦ Samples observations (usually things like request durations or response sizes) and counts them in configurable buckets. It also provides a sum of all observed values • Summary ◦ Histogram + total count of observations + sum of all observed values, it calculates configurable quantiles over a sliding time window
58. Available Languages • Official ◦ Go, Java or Scala, Python, Ruby • Unofficial ◦ Bash, C++, Common Lisp, Elixir, Erlang, Haskell, Lua for Nginx, Lua for Tarantool, .NET / C#, Node.js, PHP, Rust →
59. Prometheus Client Libaries: SpringBoot Example → → → →
60. Demo: Application metrics
61. Code Demo “Application metrics”
62. Service Discovery (Consul) Integration
63. Demo: Consul Integration
64. Service Discovery
65. Demo: Consul integration Register the services with Consul and Monitor 1 2
66. Code Demo “Consul to the rescue”
68. That’s a wrap! Question? Marco Pas Software geek, hands on Developer/Architect/DevOps Engineer @marcopas https://github.com/mpas/infrastructure-and-system-monitoring-using-prometheus