Monitoring Kubernetes Clusters with Prometheus

Transcript

1. Monitoring Kubernetes Clusters with Prometheus Fabian Reinartz Software Engineer at CoreOS @fabxc
2. Monitoring Challenges • A lot of targets to monitor • Targets constantly change • Need high-level overview (by namespace, service, …) • Need drill-down for investigation (down to pod and below)
3. Prometheus Recap • Pull-based monitoring system • Multi-dimensional data model • Handles millions of time series per instance http_requests_total{path="/home", status="200", method="GET"} 9523 http_requests_total{path="/home", status="500", method="GET"} 233 http_requests_total{path="/settings", status="200", method="GET"} 512 http_requests_total{path="/settings", status="200", method="POST"} 68
4. Powerful Querying histogram_quantile(0.99, sum by(path, le) (rate(request_latency_seconds_bucket[5m])) ) 99th percentile latency on API server operations per resource? {path="/status"} 0.012 {path="/"} 0.43 {path="/api/v1/topics/:topic"} 1.31 {path="/api/v1/topics} 0.192
5. Powerful Querying ALERT DiskWillFillIn4Hours IF predict_linear(node_filesystem_free[1h], 4*3600) < 0 Is any disk about to run full within 4 hours? 0 now -1h +4h
6. Kubernetes Integration • Always sync monitoring targets with Kubernetes API • Use meta information to enrich metrics scrape metrics API Server sync monitoring targets
7. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet etcd0 etcd1 etcd2 ... pods kube-state-metrics
8. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet etcd0 etcd1 etcd2 ... pods kube-state-metrics
9. Metrics of Kubernetes cAdvisor API server Kubelet Node node exporter etcd0 etcd1 etcd2 ... pods kube-state-metrics
10. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet etcd0 etcd1 etcd2 ... pods kube-state-metrics
11. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet etcd0 etcd1 etcd2 ... pods kube-state-metrics
12. Metrics of Kubernetes Node node exporter API server Kubelet etcd0 cAdvisor etcd1 etcd2 ... pods kube-state-metrics
13. Monitoring Challenges • A lot of targets to monitor ✓ • Targets constantly change ✓ • Need high-level overview (by namespace, service, …) ✓ • Need drill-down for investigation (down to pod and below) ✓ • AND: Make monitoring trivial to deploy & operate
14. Managed Deployments apiVersion: prometheus.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-k8s spec: replicas: 2 version: v1.3.0 Prometheus TPR defines desired Prometheus setup Operator deploys and manages Prometheus instances Operator deploy & manage Prometheus Server watch
15. github.com/coreos/kube-prometheus $ cluster-monitoring/deploy
16. Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet etcd0 etcd1 etcd2 kube-state-metrics pods
17. Monitoring as a Cluster Feature
18. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels: tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declarative definition of how to monitor a group of services Loosely coupled via labels Part of your cluster’s API
19. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels: tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Select applicable services by their labels
20. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels: tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declare where these services expose metrics
21. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels: tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Prometheus deployments include ServiceMonitors by their labels
22. Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-frontend Spec: version: v1.3.0 serviceMonitors: - selector: matchLabels: tier: frontend Prometheus deployments include ServiceMonitors by their labels
23. Service 1 Service 2 Service 3 Service 4 Service 5 ServiceMonitor 1 ServiceMonitor 2 Prometheus Operator deploy & manage Prometheus Server watch
24. Try it out! github.com/coreos/prometheus-operator github.com/coreos/kube-prometheus
25. fabian.reinartz@coreos.com @fabxc QUESTIONS? Thanks! We’re hiring: coreos.com/careers Let’s talk! #prometheus on Freenode More events: coreos.com/community LONGER CHAT? also in Berlin!