Elastic{ON} 2018 - Kubernetes, Docker, and Containers at Elastic: Monitoring, Logging, and More

Transcript

1. Infrastructure Engineer, Elastic leothrix tylerjl Kubernetes, Docker, and Containers at Elastic: Tyler Langlois Monitoring, Logging, and More Software Engineer, Beats, Elastic exekias Carlos Pérez-Aradros
2. About us Tyler Langlois Infrastructure Engineer Elastic Carlos Pérez-Aradros Software Engineer Elastic
4. 4 4 We need specific tools to track things down With containers architecture everything is a moving target
5. Beats containers toolbox 5 Modules: System, Docker and Kubernetes 1 Metadata processors 3 Autodiscover 4 Docker logs input 2
6. Metricbeat modules 6 Monitor Docker & Kubernetes
7. filebeat.prospectors: - type: docker containers.ids: - ‘*’ Parse and ship /var/lib/docker/containers/*/*.log: {"log":"INFO elasticsearch/client.go:145 Elasticsearch url:http://elasticsearch:9200\r\n","stream":"stdout","t ime":"2018-02-11T23:29:19.236692181Z"} Docker logs input Retrieve logs from Docker containers
8. Metadata processors 8 Enrich events with useful metadata to correlate logs, metrics & traces •cloud.availability_zone •cloud.region •cloud.instance_id •cloud.machine_type •cloud.project_id •cloud.provider • docker.container.id • docker.container.image • docker.container.name • docker.container.labels • kubernetes.pod.name • kubernetes.namespace • kubernetes.labels • kubernetes.annotations • kubernetes.container.name • kubernetes.container.image add_cloud_metadata add_docker_metadata add_kubernetes_metadata
9. 9 Example Metadata processors { "@timestamp": "2017-11-17T00:53:33.759Z", "message": "2017/11/07 00:53:32.804991 client.go:651: INFO Connected to Elasticsearch version 6.0.0", "kubernetes": { "pod": { "name": "filebeat-vqf85" }, "container": { "name": "filebeat" }, "namespace": "kube-system", "labels": { "k8s-app": "filebeat", "kubernetes.io/cluster-service": "true" } }, "meta": { "cloud": { "instance_id": "6959555125944564951", "instance_name": "gke-demo-default-pool-6b42dcb3-z2x7", "machine_type": "projects/865493543029/machineTypes/n1-standard-1", "availability_zone": "projects/865493543029/zones/europe-west1-b", "provider": "gce" } }, }
10. 10 add_kubernetes_metadata Metadata processors API Server add_kubernetes_metadata pod watcher Pod start/stop events Docker Logs Cont. ID Metadata update 418a913c7076 ……………… e4e4c23175cd ……………… ef997054d40c ……………… c626cfdf38614 ……………… e5563a7cb80e ……………… 73de79be045c ……………... Elasticsearch Parse Enrich
11. 11 Watch Docker events and react to changes Autodiscover metricbeat.autodiscover: providers: - type: docker templates: - condition: contains.docker.container.image: etcd config: - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379"
12. 12 Watch Docker events and react to changes Autodiscover Events API Container start/stop events Beats { "host": "10.4.15.9", "port": 2379, "docker": { "container": { "id": "13a2...d716" "name": "etcd", "image": "quay.io/coreos/etcd:v3.0.0", "labels": { "io.kubernetes.pod.name": "etcd-4dk4c", "io.kubernetes.pod.namespace": "kube- system" ... } } } } 1. autodiscover event - module: etcd metricsets: ["leader", "self", "store"] hosts: "${data.host}:2379" config template 3. var expansion 2. match condition 4. launch module - module: etcd metricsets: ["leader", "self", "store"] hosts: "10.4.15.9:2379"
13. 13 Watch Kubernetes events and react to changes Autodiscover: Kubernetes provider filebeat.autodiscover: providers: - type: kubernetes include_annotations: ["harvest"] templates: - condition: contains: kubernetes.annotations.harvest: true config: - type: docker containers.ids: - ${data.kubernetes.container.id}
14. Deployment Strategies
15. Docker deployment 15 volume mounts volume mounts Kibana Elasticsearch docker host Log files (/var/lib/docker/containers) Web Apps Services Docker API ... Networking Filebeat Metricbeat /proc filesystem
16. Kubernetes deployment (cluster) 16 Node 1 Metricbeat Filebeat Node 2 Metricbeat Filebeat Node n Metricbeat Filebeat Filebeat DaemonSet Metricbeat DaemonSet
17. Kubernetes @ Elastic
18. 18 Background What we monitor • Migration to container scheduler runtimes over traditional Cloud instances • Dogfooding! • We use GCP GKE • Stock monitoring options work, but we wanted to leverage our Stack • All container logs and metrics • Host metrics • Beats + k8s metadata Beats + Kubernetes in Practice Deployment
19. What are we interested in monitoring? 19 Some of our apps and use cases Kubernetes itself (events, unavailable pods, etc.) Why is $x burning its allocation of CPU resources? What is our current cluster capacity, and how much room to grow? Logs and metrics for dev apps, ES, Vault, Logstash, etc. “How much network traffic is our nginx ingress controller serving?”
20. 20 • We use our stock Kubernetes manifests with slight changes for our environment (helm) • RBAC and default configs all work well out-of-the box • Just pay attention where to send logs and metrics (internal vs. external Elasticsearch) This is a sample image Kubernetes Definitions Setup go.es.io/beats-k8s
21. • Coupled with emptyDir registry paths, Filebeat is performant and handles Pod restarts well • Defining pipelines at config-time is useful for extra parsing without changing container configs • Very favorable performance versus stock GKE fluentd log shippers (more on that later) Deployment Observations 21 From the field
22. The Payoff
23. Demo Time!
24. 24 Service: “beats- demo” LoadBalancer Deployment: “beats-demo“ Demo scenario: http://beats-demo.elstc.co Pod Pod ... Deployment: “mysql“ Logs Metrics Pod
25. 25 (Live!) Dashboards Example Dashboards
26. Which Nodes and Namespaces Are Busy?
27. Measuring an Individual Pod
28. Comparing Resource Utilization Across Pods
29. Comparing Resources for Multiple Pods
30. Overall Resources with Log Volume
31. Visualizing Node Utilization vs. Capacity
32. Visualizing Node Utilization vs. Capacity
33. • Use the pre-existing k8s deployment manifests to run Beats in k8s • Gets ahead of questions like Metricbeat’s ability to introspect host node metrics, Docker socket access, etc. • Pour everything into Elasticsearch and repurpose your data • Other engineers have built monitoring tools atop the metrics we’re already collecting to alert on unavailable pods with Watcher, for example (one label = monitored app) • Democratizing metrics and logs in Elasticsearch has permitted anyone to create app dashboards for their own purposes (self-serve monitoring, alerting, and more) • Textual search alongside metrics is powerful (aggregations, regexes, etc.) Lessons about Beats and Kubernetes 33
34. 34 More Questions? Visit us at the AMA
35. www.elastic.c o
36. Except where otherwise noted, this work is licensed under http://creativecommons.org/licenses/by-nd/4.0/ Creative Commons and the double C in a circle are registered trademarks of Creative Commons in the United States and other countries. Third party marks and brands are the property of their respective holders. 36 Please attribute Elastic with a link to elastic.co