New monitoring systems

A survey of new monitoring tools written in the Go programming language.

1. New monitoring systems (and why you should use experiment with them) Matt Cottingham @mattrco
2. Web Operations (Allspaw et. al.) • Great overview of approaches to ensuring the uptime of your services • Useful when I co-founded a startup
3. • Monitor systems, services and applications • Learn what is expected for a system • See trends and patterns • Discover and alert on problems Monitoring metrics
4. What’s changed since?
5. What’s changed since? • Multiple deploys per day • Application Performance Management • Breadth of IaaS/PaaS offerings • More applications (Microservices) • Anomaly detection • Containers...
6. What needs improving? • Handle ephemeral nodes • Thresholds are still a pain • Manipulating data is still hard • Make useful for others in the business?
7. Some notable Go projects Heka (by Mozilla) • Data collection and processing in use at Mozilla • Large no. input and output plugins • Logs as well as metrics • Lua sandbox for experimentation
8. Some notable Go projects Prometheus (by SoundCloud) • Tagged time series • Query DSL
9. Some notable Go projects Bosun (by Stack Exchange) • Similarities to prometheus • Run alerts against historical data! • OpenTSDB datastore
10. Some notable Go projects InfluxDB • Time series database • Based on LevelDB
11. An experiment Anode (github.com/mattrco/anode) • Setting thresholds is boring, a computer should do it • Inspired by heka and Etsy’s skyline • Thrown together in a few evenings
12. Building Anode Channels are a good fit for input, processing, output
13. Building Anode https://github.com/dgryski/go-change
14. Inspiration
15. Heka in more detail • Sandbox allows you implement certain plugin types at runtime • Change the .lua file, reload • Resources constrained • Low memory footprint (16KiB/plugin)
16. Go runtime statistics • expvars (in the standard lib) is a thing that is useful
17. Where we want to be • Adrian Cockroft’s Velocity keynote is full of good suggestions: https://vimeo. com/95064249
18. Thank you @mattrco