Share
Embed
Transcript
25 slides
-
Monitoring Challenges • A lot of targets to monitor •
Targets constantly change • Need high-level overview (by namespace, service, …) • Need drill-down for investigation (down to pod and below) -
Prometheus Recap • Pull-based monitoring system • Multi-dimensional data model
• Handles millions of time series per instance http_requests_total{path="/home", status="200", method="GET"} 9523 http_requests_total{path="/home", status="500", method="GET"} 233 http_requests_total{path="/settings", status="200", method="GET"} 512 http_requests_total{path="/settings", status="200", method="POST"} 68 -
Powerful Querying histogram_quantile(0.99, sum by(path, le) (rate(request_latency_seconds_bucket[5m])) ) 99th percentile
latency on API server operations per resource? {path="/status"} 0.012 {path="/"} 0.43 {path="/api/v1/topics/:topic"} 1.31 {path="/api/v1/topics} 0.192 -
Powerful Querying ALERT DiskWillFillIn4Hours IF predict_linear(node_filesystem_free[1h], 4*3600) < 0 Is
any disk about to run full within 4 hours? 0 now -1h +4h -
Kubernetes Integration • Always sync monitoring targets with Kubernetes API
• Use meta information to enrich metrics scrape metrics API Server sync monitoring targets -
Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet
etcd0 etcd1 etcd2 ... pods kube-state-metrics -
Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet
etcd0 etcd1 etcd2 ... pods kube-state-metrics -
Metrics of Kubernetes cAdvisor API server Kubelet Node node exporter
etcd0 etcd1 etcd2 ... pods kube-state-metrics -
Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet
etcd0 etcd1 etcd2 ... pods kube-state-metrics -
Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet
etcd0 etcd1 etcd2 ... pods kube-state-metrics -
Metrics of Kubernetes Node node exporter API server Kubelet etcd0
cAdvisor etcd1 etcd2 ... pods kube-state-metrics -
Monitoring Challenges • A lot of targets to monitor ✓
• Targets constantly change ✓ • Need high-level overview (by namespace, service, …) ✓ • Need drill-down for investigation (down to pod and below) ✓ • AND: Make monitoring trivial to deploy & operate -
Managed Deployments apiVersion: prometheus.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-k8s spec:
replicas: 2 version: v1.3.0 Prometheus TPR defines desired Prometheus setup Operator deploys and manages Prometheus instances Operator deploy & manage Prometheus Server watch -
Metrics of Kubernetes Node node exporter cAdvisor API server Kubelet
etcd0 etcd1 etcd2 kube-state-metrics pods -
Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:
tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declarative definition of how to monitor a group of services Loosely coupled via labels Part of your cluster’s API -
Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:
tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Select applicable services by their labels -
Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:
tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Declare where these services expose metrics -
Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: ServiceMonitor metadata: name: frontend labels:
tier: frontend spec: selector: matchLabels: tier: frontend endpoints: - port: web path: /metrics interval: 30s Prometheus deployments include ServiceMonitors by their labels -
Operator (continued) apiVersion: monitoring.coreos.com/v1alpha1 kind: Prometheus metadata: name: prometheus-frontend Spec:
version: v1.3.0 serviceMonitors: - selector: matchLabels: tier: frontend Prometheus deployments include ServiceMonitors by their labels -
Service 1 Service 2 Service 3 Service 4 Service 5
ServiceMonitor 1 ServiceMonitor 2 Prometheus Operator deploy & manage Prometheus Server watch -
[email protected] @fabxc QUESTIONS? Thanks! We’re hiring: coreos.com/careers Let’s talk! #prometheus
on Freenode More events: coreos.com/community LONGER CHAT? also in Berlin!