Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Fool-proof K8s dashboards for sleep-deprived on...

David
May 23, 2019

Fool-proof K8s dashboards for sleep-deprived on-calls

Being on-call is hard. Your Kubernetes dashboards should not make it even harder. This talk gives you a couple of best practices for effective dashboarding. I've wrapped this all into a Dashboarding Maturity Model (DMM) to give you some indicators on how to get your dashboarding practices to the next level.

David

May 23, 2019
Tweet

More Decks by David

Other Decks in Technology

Transcript

  1. I’m David Working on Explore, Prometheus, and Loki at Grafana

    Labs Previously: Unifying Metrics/Logs/Traces at Kausal, Work on WeaveScope [email protected] Twitter: @davkals
  2. Cognitive load In which direction do I have to pull

    the little lever to open the metro door?
  3. On-call - Good on-call is debugging and follow-up, improving things

    for the rest. - Bad on-call is mostly incident response where every minute counts
  4. Dashboarding maturity levels Low Default state (no strategy) Medium Managing

    use of methodical dashboards High Optimizing use, consistency by design
  5. Dashboarding maturity levels Low No strategy (default state) Medium Managing

    the use of methodical dashboards High Optimizing use, consistency by design
  6. Medium maturity: Methodical dashboards - USE method for resources: For

    each resource measure utilization, saturation, errors - RED method for services: For each service measure request and error rate, and duration - Your own method
  7. Medium maturity: Hierarchical dashboards - Summary views with aggregate queries

    - Queries have breakdown by next level - Tree structure reflecting the k8s hierarchies
  8. Medium maturity: Service hierarchies - RED method - One row

    per service - Row order reflects data flow
  9. Medium maturity: Expressive charts - Meaningful use of color -

    Normalize axis where you can - Understand the underlying metrics
  10. Medium maturity: Directed browsing - Template variables make it harder

    to “just browse” - Most dashboards should be linked to by alerts - Browsing is directed (drill-down)
  11. Medium maturity: Managing dashboards - Version controlled dashboard sources -

    Currently by copy/pasting JSON - RFC in our design doc
  12. Dashboarding maturity levels Low Default state (no strategy) Medium Managing

    use of methodical dashboards High Optimizing use, consistency by design
  13. High maturity: Optimizing use - Actively reducing sprawl - Regularly

    reviewing existing dashboards - Tracking use (upcoming feature: meta-analytics)
  14. High maturity: Consistency by design - Use of scripting libraries

    to generate dashboards - grafonnet (Jsonnet) - grafanalib (Python) - Consistent attributes and styles across all dashboards - Smaller change sets g.dashboard('Cluster').addRow( g.row('CPU').addPanel( g.panel('CPU Utilisation') + g.queryPanel('node:cluster_cpu_utilisation:ratio') + g.stack + { yaxes: g.yaxes({ format: 'percentunit', max: 1 }) }, ).addPanel( g.panel('CPU Saturation (Load1)') + g.queryPanel(||| node:node_cpu_saturation_load1: / scalar(sum(min(kube_pod_info) by (node))) |||) + g.stack + { yaxes: g.yaxes({ format: 'percentunit', max: 1 }) }, ) )
  15. High maturity: Use of mixins or other peer-reviewed templates Prometheus

    Monitoring Mixins Talk at PromCon 2018 by Tom Wilkie https://www.youtube.com/watch?v=GDdnL5R_l-Y
  16. Future workflow: Dashboard as code - Live edit JSON and

    preview dashboards - Live edit Jsonnet or Python sources and preview in browser - Open PR directly from Grafana
  17. Dashboarding maturity levels Low No strategy (default state) - Everyone

    can modify - Duplicate used regularly - One-off dashboards - No version control - Lots of browsing Medium Managing use of methodical dashboards - prevention of sprawl - use of template variables - methodical dashboards - hierarchical dashboards - expressive charts - version control - directed browsing High Optimizing use, consistency by design - active sprawl reduction - use of scripting libraries - use of mixins - no editing in the browser - browsing is the exception