visualization of Docker metrics

Docker メトリクスの可視化 in Japanese

Transcript

1. VISUALIZATION OF DOCKER METRICS 2015/12/09 by janus_wel
2. WHO? 高木健介 janus_wel archtect/programmer for full-stack
3. PURPOSE OF VISUALIZATION ?
4. TO CHECK LIFE OR DEATH ? should be covered by orchestration ex) consul
5. TO USE RESOURCES EFFICIENTLY !! figure out bottlenecks decision a number of containers on a machine
6. TO MAKE DOCUMENTS for customers for executives
7. TO FOLLOW THE PDCA CYCLE figure out influences of changes
8. COLLECTING METRICS USUALLY is important to trace back
9. TOOLS Docker remote API InfluxDB
10. DOCKER REMOTE API https://docs.docker.com/engine/reference/api/docker_remote_
11. REST LIKE API The API tends to be REST, but for some complex commands, like attach or pull, the HTTP connection is hijacked to transport STDOUT, STDIN, and STDERR.
12. AVAILABILITY maybe, all operations in commands are available. https://docs.docker.com/engine/reference/api/docker_remote_
13. FOR EXAMPLE List containers Create a container Start a container Stop a container Remove a container List Images Build image from a Dockerfile Create an image Display system-wide information Show the docker version information ...
14. REQUIREMENTS needs some settings for access control $DOCKER_OPTS in /etc/init/docker.conf on Ubuntu $OPTIONS in /etc/sysconfig/docker on other Linux if 192.0.2.1 is a client, D O C K E R _ O P T S = " - H 1 9 2 . 0 . 2 . 1 : 2 3 7 6 - H u n i x : / / / v a r / r u n / d o c k e r . s o c k " O P T I O N S = " - H 1 9 2 . 0 . 2 . 1 : 2 3 7 6 - H u n i x : / / / v a r / r u n / d o c k e r . s o c k " https://docs.docker.com/engine/userguide/basics/#bind- docker-to-another-host-port-or-a-unix-socket
15. NOTE make sure to hear from unix domain socket. O P T I O N S = " - H 1 9 2 . 0 . 2 . 1 : 2 3 7 6 - H u n i x : / / / v a r / r u n / d o c k e r . s o c k " # ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ or, docker commands are not available.
16. FOR DOCKER MACHINE machines created by docker-machine needs https by default, in your ~/.docker/machine/machines/(machine name)/ cert.pem key.pem this path is set by "$(docker-machine env (machine name))" as $DOCKER_CERT_PATH
17. INFO w g e t h t t p : / / ( m a c h i n e i p ) : 2 3 7 6 / i n f o - O - - q | j q . for docker-machine w g e t \ - - n o - c h e c k - c e r t i f i c a t e \ - - c e r t i f i c a t e = $ { D O C K E R _ C E R T _ P A T H } / c e r t . p e m \ - - p r i v a t e - k e y = $ { D O C K E R _ C E R T _ P A T H } / k e y . p e m \ h t t p s : / / $ ( d o c k e r - m a c h i n e i p ( m a c h i n e n a m e ) ) : 2 3 7 6 / i n f o \ - O - - q | j q . "curl" can't handle non-checking local cert, so use "wget"
19. STATS API to get container's metrics, use stats API GET /containers/(id)/stats https://docs.docker.com/engine/reference/api/docker_remote_ container-stats-based-on-resource-usage
21. INFLUXDB https://influxdb.com/
23. POINTS schemaless SQL like query REST like interface
24. SCHEMALESS there is no need to define somethings when writing data, InfluxDB needs only TABLE NAME and VALUE these are optional timestamp tags https://influxdb.com/docs/v0.9/concepts/schema_and_data_la
25. SQL LIKE QUERY to read S E L E C T * F R O M c p u _ l o a d _ s h o r t W H E R E h o s t = s e r v e r 0 1 A N D r e g i o n = u s - w e s t - - ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ ^ - - t a b l e n a m e t a g s https://influxdb.com/docs/v0.9/query_language/data_explor
26. REST LIKE INTERFACE # C R E A T E D A T A B A S E c u r l - G h t t p : / / l o c a l h o s t : 8 0 8 6 / q u e r y - - d a t a - u r l e n c o d e " q = C R E A T E D A T A B A S E m y d b " # I N S E R T c u r l - i - X P O S T ' h t t p : / / l o c a l h o s t : 8 0 8 6 / w r i t e ? d b = m y d b ' - - d a t a - b i n a r y ' c p u _ l o a d _ # S E L E C T c u r l - G ' h t t p : / / l o c a l h o s t : 8 0 8 6 / q u e r y ? d b = m y d b ' - - d a t a - u r l e n c o d e " q = S E L E C T v a l u https://influxdb.com/docs/v0.9/guides/writing_data.html https://influxdb.com/docs/v0.9/guides/querying_data.html
27. SYSTEM pull archtecture collect metrics minutely run with various OSs Linux Windows others can run Docker for now, test only in Linux...
29. METRICS COLLECTOR get metrics from Docker remote API and insert them InfluxDB in Ruby requires gem "clockwork" runs a collection minutely
30. VISUALIZE in HTML and JavaScript use chart.js use some EcmaScript 2015 features "let" statement Promise Object.assign http://www.chartjs.org/ now available with only Google Chrome
31. DEMO http://52.193.71.92/
32. METRICS legend description type rx_bytes received bytes counter tx_bytes transferred bytes counter total_cpu_usage cpu usage time % counter
33. SOURCES https://github.com/januswel/docker-metrics
34. PROBLEMS load stats API is heavy pull architecture calling API is proportional to number of containers handling mass containers id is not human-friendly
35. PUSH ARCHITECTURE AND AGENT to collect with low load, we need inspect /sys c a t / s y s / f s / c g r o u p / m e m o r y / d o c k e r / < c o n t a i n e r - i d > / m e m o r y . s t a t c a t / s y s / f s / c g r o u p / c p u a c c t / d o c k e r / < c o n t a i n e r - i d > / c p u a c c t . s t a t pull architecture is unfit to cat local files agent is needed https://docs.docker.com/engine/articles/runmetrics/
36. AFTER increase collectable metrices I don't know Docker metrics means what is... agent-ize collector