What’s beyond DORA metrics?

What’s beyond DORA metrics? Michael Stahnke VP Engineering at Flox
(flox.dev) @stahnma

flox.dev

@stahnma Edu Career 9 Years 8 Years 4 Years 2
Years

@stahnma

@stahnma We are here.

@stahnma Availability  Cost for Downtime COGS  Cost for Services in
support of Gross Margin Security   Attach Rate and dollar value for compliance docs and security questionnaires Developer Productivity  Hiring Tradeoffs, time to value, throughput Developer Sentiment Developer Adoption

@stahnma Developer Productivity  Hiring Tradeoffs, time to value, throughput

We are here. @stahnma

@stahnma Do I believe this?

@stahnma

@stahnma 1. …

@stahnma An approach to metrics

@stahnma

@stahnma Gathering Data

@stahnma Controlled

@stahnma Targeted

@stahnma Gathering Make observations like “interesting” or “what do we
think that means?” Controlled Has an upper and lower bound. If out of bounds, an “incident” is declared. Sometimes called a KPI. Targeted You are actively doing projects/ work to change and improve this measurement. Sometimes called an OKR.

@stahnma Where have I found my orgs historically?

We are here. @stahnma

@stahnma I don’t feel elite.

@stahnma Why don’t I feel elite?

@stahnma These metrics lack context

@stahnma They’re throughput oriented

@stahnma

@stahnma Incomplete analysis of “good”

@stahnma Deployment Frequency is a vanity metric after it’s in
control

@stahnma MTTR is unhelpful for improvements in most scenarios

@stahnma Lead Time to Change is extremely context dependent

@stahnma Change failure rate treats all failures the same

@stahnma cyn·i·cism    noun    an inclination to believe that
people are motivated purely by self-interest; skepticism.

@stahnma Hating is easier than trying.

@stahnma There is so much good stuff from DevOps Metrics
and reports

@stahnma There is value in collecting those

@stahnma All models are flawed. Hopefully this model has fewer
flaws.

@stahnma What truly is elite?

@stahnma Moving fast with confidence.

@stahnma Move from IC reported to aggregate

@stahnma Move from reported to derived

@stahnma

@stahnma What are the categories of things I care about?

@stahnma Is engineering throughput good?

@stahnma Is engineering throughput good? Partially covered by existing metrics

@stahnma Is that throughput meaningful?

@stahnma Is that throughput meaningful? Not really covered by existing
metrics

@stahnma Metric Thoughts Layer Cake Quality People Throughput

@stahnma BLUF

@stahnma Mental Model for Metrics Baseline DevOps Metrics I don’t
feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions People Metric Metric Metric Themes

@stahnma There is one theme to success using devops practices

@stahnma This is backed by 6 Sigma

@stahnma This is backed by Lean

@stahnma This is backed by The Toyota Way

@stahnma This is backed by basic logic

@stahnma This is backed by the 2018 report

@stahnma And all other SODORs

@stahnma Reduce variance

@stahnma Throughput Measurements

@stahnma deployments / week

@stahnma Does adding more developers make things go faster?

@stahnma deploys / dev / week

@stahnma deploys / dev / week Using merges instead of
deploys can be a proxy for this in some scenarios

@stahnma 1. … @stahnma

@stahnma ratio of distraction action

@stahnma ratio of distraction action Sometimes called “Investment ratios”

@stahnma narrow vs wide work

@stahnma 1. … @stahnma

@stahnma People

@stahnma Are our systems easy to work on?

@stahnma

@stahnma fungible engineering?

@stahnma

@stahnma normalization

@stahnma % of code provided by the platform.

@stahnma

@stahnma How quickly are they productive?

@stahnma 5th contribution (PR)

@stahnma deploys / dev / week

@stahnma Quality

@stahnma successful deployments / week

@stahnma rollbacks (or roll forwards)

@stahnma understanding changes happening all around you

@stahnma

@stahnma Metric Thoughts Layer Cake Quality People Throughput

@stahnma Mental Model for Metrics Baseline DevOps Metrics I don’t
feel elite. Move Quickly with Confidence Throughput Quality Normalization Fungibility Rollbacks (or forwards) Deployments per Week Deploys / Dev / Week Successful deploys Narrow vs Wide Time to 5th PR Changes Understood Start Status Questions People Metric Metric Metric Themes

@stahnma Summary • What do you want your metrics to
tell you? • Can you correlate the movement with intentional action and improved outcomes? • Do you have baseline metrics already? • How have you joined these up with product management? • Deploys / Dev / Week is the _one_ I would start with • Reduce _if_ statements

@stahnma Twitter’s Engineering Effectiveness Let 1,000 Flowers Bloom - http://www.gigamonkeys.com/
flowers/ The SPACE of developer Productivity. Research paper from Nicole Forsgren, GitHub Margaret-Anne Storey, University of Victoria Chandra Maddila, Thomas Zimmermann, Brian Houck, and Jenna Butler, Microsoft Research https://queue.acm.org/detail.cfm?id=3454124 GitLab MR Rate https://about.gitlab.com/handbook/engineering/merge- request-rate/ This deck is at https://speakerdeck.com/ stahnma/whats-beyond-dora-metrics

@stahnma We’re hiring We’re post-geographic.

flox.dev

What’s beyond DORA metrics?

What’s beyond DORA metrics?

More Decks by Michael Stahnke

Featured

Transcript