Secrets to Monitor Kubernetes Workloads

Monitoring systems remain a critical challenge in the ever-evolving landscape of Kubernetes workloads. It is no longer a second thought and goes hand in hand with development and deployment. Throw in the challenge of data growth, high cardinality and the complexities multifold, not just in data growth management but also in fatigue and cost.

Join us as we delve into the trenches of Kubernetes observability, uncovering the secrets to monitor your workloads masterfully using Open Standards. Our journey will explore advanced techniques, addressing real-world challenges people encounter daily with trade-offs from the trenches and lessons learned while building one of the biggest monitoring platforms.

Transcript

1. Supported By Secrets to Monitor Kubernetes Workloads Prathamesh Sonpatki Developer Evangelist, Last9
2. whoami Supported By ഹേലാ
3. whoami - Prathamesh Sonpatki - Developer turned Developer Evangelist at Last9 - SRE Stories - O11y.wiki - Twitter, Linkedin Supported By
4. Supported By last9.io/friends
5. Agenda - Why should you care? - Why “Monitoring”? - Prometheus and Kubernetes - Challenges of Monitoring Kubernetes Workloads - Some Cool Stuff Supported By
6. Why should you care? Kubernetes is de-facto standard managing and deploying modern microservices based apps. Supported By
7. Why should you care? A big box of interconnected components. Supported By
8. Why should you care? - Fun to learn new things! Supported By
9. What is Monitoring? Supported By https://last9.io/blog/radar-and-black-boxes-for-software-observability/ vs.
10. What is Monitoring? Supported By
11. Monitoring Kubernetes Workloads Who is monitoring? - DevOps / SRE / Operators - App Developers - Product - Business Stakeholders Supported By
12. Monitoring Kubernetes Workloads When they are monitoring? - War Time - Peace Time Supported By
13. Ideal Monitoring Wishlist* Supported By Pre-Processing Pipelines Instrumentation Long Term Storage Faster Access Standard Query Interface Standard Dashboards Workload Isolation Relevant Context Automation Coverage Signal vs. Noise Ingestion Storage Query Alerting
14. Monitoring Kubernetes Workloads (at bare minimum…) - Monitoring Kubernetes itself Supported By - Monitoring Apps, Services, Jobs running in Kubernetes
15. Monitoring Kubernetes itself - Kubernetes exposes Metrics in OpenMetrics format for different components https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/ Supported By
16. Monitoring Kubernetes itself - Kubernetes exposes Metrics in OpenMetrics format for different components https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/ Supported By
17. Monitoring Kubernetes itself - Kubernetes exposes Metrics in OpenMetrics format for different components https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/ Supported By
18. Monitoring Kubernetes itself - Kubernetes exposes Metrics in OpenMetrics format for different components https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/ Supported By
19. Monitoring Kubernetes itself - Kubernetes exposes Metrics in OpenMetrics format for different components https://kubernetes.io/docs/tasks/debug/debug-cluster/resource-usage-monitoring/ Supported By
20. Monitoring Apps, Services on Kubernetes Supported By
21. Monitoring Apps, Services on Kubernetes Supported By - The challenge of Service Discovery and Ephemeral resources
22. Monitoring Apps, Services on Kubernetes Supported By
23. Monitoring Kubernetes using Prometheus Supported By
24. Prometheus ❤ Kubernetes - Kubernetes was inspired from Borg, Google’s container orchestration system - Prometheus was inspired from Borgman, a system to monitor services on Borg. - Both are CNCF graduated projects. - Supported By
25. Monitoring Kubernetes workloads using Prometheus Supported By https://sysdig.com/blog/kubernetes-monitoring-prometheus/
26. Monitoring Kubernetes workloads using Prometheus Supported By OpenMetrics Format Text Exposition Format Pull based model Instrumentation Multidimensiona l data model Long Term Storage PromQL Grafana Built-in visualization Standard Dashboards AlertManager Silencers Inhibit Rules Notification Destinations Service Discovery Standard K8S Metrics Prometheus Exporters Ingestion Storage Query Alerting
27. Challenges in monitoring Kubernetes Workloads Supported By ? Instrumentation ? ? ? ? Ingestion Storage Query Alerting
28. Let’s roll this Supported By
29. Setup & Instrumentation Supported By
30. Setting up Kubernetes Monitoring using Prometheus - Avoid Google Search - Just use “kube-prometheus” Supported By
31. Setting up Kubernetes Monitoring using Prometheus Supported By
32. Setting up Kubernetes Monitoring using Prometheus Supported By https://training.promlabs.com/t raining/prometheus-and-kuber netes/prometheus-operator-int roduction/operator-architectur e
33. Challenges in monitoring Kubernetes Workloads Supported By ? Instrumentation ? ? ? Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
34. Ingestion Supported By
35. Ingestion - Ingesting large volumes of data is rarely a problem. - As long as reads are not significant. - As long as system is not already primed. - As long as storage is not for long durations. Supported By
36. Ingestion The High Cardinality Challenge 4 (Clusters) x 1000 (Service) x 3 (Environment) x 10,000 (Pods) = 12,000,000 🙀 Supported By https://www.foomo.org/blog/prometheus- cardinality-issues
37. Solving High Cardinality Downsampling Supported By
38. Solving High Cardinality Drop labels Supported By
39. It is difficult to make predictions, especially about the future Supported By
40. Challenges in monitoring Kubernetes Workloads Supported By Drop ✅ Relabel ✅ Rename ✅ Cardinality 😭 Compute 😭 Cost 😭 Instrumentation ? ? ? Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
41. Storage Supported By
42. Storage Supported By - Millions of metrics but mostly unused. - Data exists but not accessible when needed. - Lack of workload isolation - Alerting needs fresh data, SLOs need data on longer horizon. - Can do remote write / Thanos / Levitate
43. Challenges in monitoring Kubernetes Workloads Supported By Drop ✅ Relabel ✅ Rename ✅ Cardinality 😭 Compute 😭 Cost 😭 Instrumentation PV ✅ Storage Cost 😭 Remote Write ✅ ? ? Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
44. Query Supported By
45. Query Supported By - You are fine as long as you are not reading. - Storage - Single Point of Failure
46. Query Supported By - You are fine as long as you are not reading. - Storage - Single Point of Failure - Compute cost skyrockets. - Solution: Limit instrumentation / Limit query range. - Lack of traffic shaping.
47. Challenges in monitoring Kubernetes Workloads Supported By Drop ✅ Relabel ✅ Rename ✅ Cardinality 😭 Compute Cost 😭 Instrumentation PV ✅ Storage Cost 😭 Remote Write ✅ PromQL ✅ Standard Dashboards ✅ Concurrent Access 😭 Compute Cost 😭 ? Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
48. Alerting Supported By
49. Alerting Supported By - Fatigue - Signal vs. Noise - Historical Trends
50. Alerting Supported By - Thresholds - Seasonality
51. Alerting Supported By Sane Defaults https://samber.github.io/awesome-prometheus-alerts/rules#kubernetes
52. Alerting Supported By Dark Side of Sane Defaults - High Cardinality Labels in alerts and dashboards - Latent Dashboards - Invalid Alert Rules
53. Challenges in monitoring Kubernetes Workloads Supported By Drop ✅ Relabel ✅ Rename ✅ Cardinality 😭 Compute Cost 😭 Instrumentation PV ✅ Storage Cost 😭 Remote Write ✅ PromQL ✅ Standard Dashboards ✅ Concurrent Access 😭 Compute Cost 😭 AlertManager ✅ Destinations ✅ Good Defaults 😐 No anomaly 😭 Basic Viz 😭 Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
54. Security Supported By
55. Security - Prom Operator runs on the cluster itself! - Automatically gets right creds - Can reach targets without extra networking permissions Supported By
56. Cool Stuff Supported By
57. Reduce Kubernetes Metrics - Drop everything not used by Dashboards and Alerts. - Mimirtool Supported By
58. Autoscale based on metrics - Did you know, you can connect HPA to Metrics? - Use prometheus-adapter Supported By
59. Streaming Aggregations - Run Aggregations before data is stored to reduce cardinality. - Ignore pods, instances when not needed. Supported By
60. Challenges in monitoring Kubernetes Workloads Supported By Drop ✅ Relabel ✅ Rename ✅ Cardinality 😭 Compute Cost 😭 Instrumentation PV ✅ Storage Cost 😭 Remote Write ✅ PromQL ✅ Standard Dashboards ✅ Concurrent Access 😭 Compute Cost 😭 AlertManager ✅ Destinations ✅ Good Defaults ✅ No anomaly 😭 Basic Viz 😭 Operator ✅ Service Discovery ✅ Service Monitoring ✅ Ingestion Storage Query Alerting
61. Supported By നന്ദി llast9.io/levitate-tsdb Prathamesh Sonpatki Developer Evangelist, Last9