Monitor Docker Containers on CoreOS cluster #monitoringcasual

http://www.zusaar.com/event/9807003

Transcript

1. Monitor Docker Containers on CoreOS cluster
2. I’m Taichi Nakashima @deeeet tcnksm https://www.flickr.com/photos/unforgiven/9278027165
7. What is difference? Traditional stack vs. Containers
8. Physical (Machines) Virtual Virtual Service A Service B Service C Service D
9. Physical (Machines) Virtual Virtual Container Container Container Container Service A Service B Service C Service D
10. Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service
11. Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service
12. For a single host monitoring cAdvisor For a cluster scale monitoring Heapster Monitoring by Yourself Have your own monitoring system
14. Just running dockerized cAdvisor container Collect all container metrics on a host Just access to :8080 in your browser Provide Web UI cAdvisor
15. Run cAdvisor container $ docker run \ --volume=/:/rootfs:ro \ --volume=/var/run:/var/run:rw \ --volume=/sys:/sys:ro \ --volume=/var/lib/docker/:/var/lib/docker:ro \ --publish=8080:8080 \ --detach=true \ --name=cadvisor \ google/cadvisor:latest
17. Collect cAdvisor metrics from cluster member, it’s used in Kubernetes Enables cluster wide monitoring of containers Draw graph by Grafana Support InfluxDB backend Heapster
18. Container
19. Container
20. Container Heapster
21. Container Heapster 
23. Monitoring by Yourself Have your own monitoring system Monitoring as a Service Use external service
27. But No README and No document… stanaka/mackerel-docker Mackerel Not Support container specific feature No container specific monitoring
28. New Relic Dockerized collector agent but only for host metrics not for containers, To monitor each container we need to install it on each our docker image johanneswuerbach/newrelic-sysmond Not Support container specific feature No container specific monitoring
29. DataDog Dockerized collector agent, just run docker container DataDog/docker-dd-agent Support container specific feature !! Container metrics, Tagging, Lifecycle of container, etc
30. DataDog Container feature What is good point and why ?
32. DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events
33. DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start
34. Run dd-agent container $ docker run \ --privileged \ --name dd-agent \ -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY={your_api_key_here} \ datadog/docker-dd-agent
35. Container dd-agent
36. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
37. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
38. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
39. To dd-agent container on CoreOS cluster $ fleetctl start dd-agent.service Run dd-agent.service by fleet
40. DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to explore what you want
41. DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Help understand unexpected value
45. New containers are created
46. DataDog container feature Agent tags by docker container name and its image name (by default) Tagging All containers on a host If you run 1 dd-agent container, it monitors all containers in the host It collects each container’s CPU, memory, network I/O and disk I/O (General) Lifecycle monitoring Agent also monitor container create, start, stop, destroy events Easy to start Easy to explore what you want Help understand unexpected value
47. Requirement for container monitoring Common part of recent trend
48. Container Heapster 
49. Container dd-agent
50. Install agent each container is not good idea, keep container simple ! All containers in a host by 1 agent Requirement for container monitoring Containerized Agent Container only approach is Docker-way (CoreOS)
51. Manage secret values on distributed KVS Save API token on etcd/consul Extra edition
52. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`YOUR_API_KEY` \ datadog/docker-dd-agent" [X-Fleet] Global=true
53. xordataexchange/crypt
54. Generate pub and sub key $ gpg2 --gen-key # generate pub.gpg and secret.gpg
55. Save json value on etcd by crypt $ cat <<EOF > config.json {"test": "passw0rd"} EOF $ crypt set -keyring=pub.gpg /app/config config.json
56. Get value withtout crypt and secret-key $ etcdctl get /app/config wcBMA0OL+oKDi4zdAQgAh7iKVASBZvvX6WiiLPYSZgAbhYDhZyVGqX +uK2Bc1plC/mYkqw/n3FXyL+ZC0ISdK9Hdqv6HpCthnMHmBCfhPAjV4 DsrXKWO7TP0AYTxUPMxX9sIiTzrLTJGb73134Z6l0z0Ocj2dEuhyAt5u 3cucKkQb3CWGyuhM7C02aTeJoPjIkqi3agAizQn0uwcurSONpmCkArq33 3579iHZv42Xnr+1Dq4CkcDG9OYPyKcoixOvvW9OpB1E
57. Get value from etcd by crypt and key $ crypt get -secret-keyring secret.gpg /app/config {"test":"passw0rd"}
58. To dd-agent container on CoreOS cluster [Unit] … [Service] TimeoutStartSec=0 ExecStartPre=-/usr/bin/docker kill dd-agent ExecStartPre=-/usr/bin/docker rm dd-agent ExecStartPre=/usr/bin/docker pull datadog/docker-dd-agent ExecStart=/usr/bin/bash -c \ "/usr/bin/docker run --privileged --name dd-agent -h `hostname` \ -v /var/run/docker.sock:/var/run/docker.sock \ -v /proc/mounts:/host/proc/mounts:ro \ -v /sys/fs/cgroup/:/host/sys/fs/cgroup:ro \ -e API_KEY=`crypt get -secret-keyring /etc/secret.gpg /ddapikey` \ datadog/docker-dd-agent" [X-Fleet] Global=true
59. @deeeet