Berlin 2013 - collectd Workshop - Florian Forster

Transcript

1. collectd Tracking custom metrics Florian “octo” Forster at monitorama.eu 2013
2. Agenda • About myself and collectd • collectd data model • The exec plugin • The StatsD plugin • Other possibilities
3. • Florian “octo” Forster ◦ Twitter: @flocto ◦ Github: octo • Started the collectd project in 2005 (while at university) • Still maintaining collectd as a spare time project About myself
4. About collectd [1/3] • Free / open-source software ◦ Daemon: GPLv2 ◦ Plugins: Various free licenses (GPL, BSD, MIT) • Continuously running daemon • Specific logic in plugins ◦ Over 100 plugins, from AMQP to ZFS.
5. About collectd [2/3] • “Output” options include: ◦ Graphite ◦ MongoDB ◦ Riemann ◦ RRDtool / RRDCacheD • Various networking options
6. About collectd [3/3] • Website: http://collectd.org/ • Twitter: @collectd • Github: collectd • Hackathon tomorrow at Michelberger Hotel.
7. Agenda • About myself and collectd • collectd data model • The exec plugin • The StatsD plugin • Other possibilities
8. Data model [1/5] • Four “data source types” • Most important: ◦ GAUGE Absolute value, e.g. temperature ◦ DERIVE Counting values, e.g. number of requests served • Special case: ◦ COUNTER
9. Data model [2/5] Strict naming schema: • Hostname • Plugin • Type Plugin and type can have an “instance” each. Examples: • example.com • df – root • df_complex – free
10. Data model [3/5] • The Type is defined in the types.db file (types.db(5) manual page). • Specifies the “data source type”, i.e. GAUGE or DERIVE. • Specifies minimum and maximum values.
11. Data model [4/5] • Plugin instance is used to differentiate between “devices” the plugin can query. ◦ For example interfaces (“eth0”), CPU number (“0”), mount point (“var-spool”). • Type instance is used to to differentiate between multiple values of the same type. ◦ For example, number of bytes used, number of bytes reserved, number of bytes free on a file system.
12. Data model [5/5] • Notation: • Examples: ◦ example.com/df-root/df_complex-free ◦ example.com/cpu-0/cpu-wait ◦ example.com/memory/memory-used host/plugin-instance/type-instance host/plugin/type
13. Agenda • About myself and collectd • collectd data model • The exec plugin • The StatsD plugin • Other possibilities
14. The exec plugin • Executes arbitrary scripts / binaries • Scripts print metrics to STDOUT • Scripts print error messages to STDERR
15. The exec plugin collectd exec plugin Custom script STDOUT / STDERR
16. The exec plugin Configuration LoadPlugin "exec" <Plugin "exec"> Exec "gandalf:wizard" "/usr/lib/collectd/exec/magic.sh" # Exec "user:group" "/path/to/my/script" </Plugin>
17. The exec plugin First version #!/bin/sh level=`magic-ctl --level` echo "PUTVAL \"example.com/exec/gauge-magic\" N:${level}"
18. The exec plugin • Performance? • Problem: Forking is expensive. • Solution: Let scripts loop and keep them running.
19. The exec plugin Second version #!/bin/sh while sleep 10; do level=`magic-ctl --level` echo "PUTVAL \"example.com/exec/gauge-magic\"" \ "interval=10 N:${level}" done
20. The exec plugin • Automation? • Problem: Hard-coded hostname and interval must match collectd configuration. • Solution: Use environment variables set by collectd: COLLECTD_HOSTNAME and COLLECTD_INTERVAL.
21. The exec plugin Final version #!/bin/sh interval=${COLLECTD_INTERVAL:-10} hostname=${COLLECTD_HOSTNAME:-localhost} while sleep ${interval}; do level=`magic-ctl --level` echo "PUTVAL \"${hostname}/exec/gauge-magic\" " \ "interval=${interval} N:${level}" done
22. The exec plugin Caveat: Buffering • collectd uses pipes. • Many environments switch to (block) buffered I/O in this case. → Switch to line buffering or disable buffering.
23. Agenda • About myself and collectd • collectd data model • The exec plugin • The StatsD plugin • Other possibilities
24. The StatsD plugin • Opens a UDP socket and waits for asynchronous “events”. • Implements the “StatsD” network protocol. ◦ Wire compatible to Etsy's reference implementation. ◦ Lots of tooling available. • Very efficient.
25. The StatsD plugin Configuration LoadPlugin "statsd" <Plugin "statsd"> DeleteCounters true DeleteSets false TimerPercentile 95 </Plugin>
26. The StatsD plugin The StatsD protocol: • Four metric types: Counter, Timer, Gauge and Set. • Mapped to collectd “types” derive, latency, gauge and objects.
27. The StatsD plugin The StatsD protocol: Counter • Increment “foo” by 3: ◦ foo:3|c • Mapped to: ◦ <host>/statsd/derive-foo
28. The StatsD plugin The StatsD protocol: Timer • Operation “bar” took 3117 ms: ◦ bar:3117|ms • Mapped to: ◦ <host>/statsd/latency-bar-average ◦ <host>/statsd/latency-bar-percentile-95 ◦ …
29. The StatsD plugin The StatsD protocol: Gauge • Set “qux” to 42: ◦ qux:42|g • Increase “qux” by 23: ◦ qux:+23|g • Mapped to: ◦ <host>/statsd/gauge-qux
30. The StatsD plugin The StatsD protocol: Set • Add “bird” to set “animals”: ◦ animals:bird|s • Mapped to: ◦ <host>/statsd/objects-animals
31. Agenda • About myself and collectd • collectd data model • The exec plugin • The StatsD plugin • Other possibilities
32. Other possibilities Network protocol: • Send metrics in the “binary network protocol”. • Various client libraries available. • Virtually no coupling.
33. Other possibilities UnixSock plugin • Listens on a UNIX domain socket and waits for commands, including PUTVAL. • Client libraries available. • Loose coupling.
34. Other possibilities Java, Perl and Python plugins • Embed the VM / interpreters into the daemon (like mod_perl and friends). • “Best of two worlds” (YMMV) • Tight coupling
35. Other possibilities C based plugins: • Play with the cool kids, drink the Kool-Aid and feel the power!
36. Thank you! Please ask some questions now! ;-) Hackathon, tomorrow (September 21) at Michelberger Hotel, starting at 9:00.