DevOps Fom a Different Data Set (short version)

What 30 million workflows reveal about high performing teams. Michael Stahnke has been an author on Puppet's groundbreaking State of DevOps Report, but as he moved to CircleCI, he wanted to see the differences in the data when comparing survey gathered metrics vs system metrics from 30 million workflows on CircleCI.

Transcript

1. 1 DevOps from a 
 Different Data Set 
 (the short version) What 30 million workflows reveal about high performing teams Michael Stahnke VP Platform Engineering @stahnma
2. 2 @stahnma
3. 3 @stahnma
5. Performance derived vs performance described 5 @stahnma
6. 6 @stahnma
7. 7 State of DevOps Report Metrics Deployment Frequency Lead time to Change Change Failure Rate MTTR Description when mapping to CI How often you initiate a pipeline Pipeline duration Pipeline failure rate Time from red to green Metric Mapping Metrics Throughput Duration Success Rate Recovery Time @stahnma
8. 8 @stahnma
9. Most projects are not deploying dozens of times per day 9 @stahnma
10. “Primary application or service you work on” 10 @stahnma
11. How long does it take to get results? 11 @stahnma
12. 12 @stahnma
13. Half of all builds finish in under 4 minutes 13 @stahnma
14. How often does your pipeline complete with a green status? 14 @stahnma
15. 15 @stahnma
16. Time a pipeline sits in a failure state 16 @stahnma
17. 17 Percentile 2020 Value 2019 Value 5p 2.06 min 2.83 min 50p 55.11 min 52.5 min 90p 39 hours 47 hours 95p 3.4 days 3.93 days Mean 14.85 hours 16.61 hours Recovery Time @stahnma
18. Quick Recovery Time can be from multiple contributors running in parallel 18 @stahnma
19. Recovery Times have improved year over year 19 @stahnma
20. 20 What development practices definitively work?
21. Success Rate does not correlate with company size 21 @stahnma
22. Duration is longest for teams of one 22 @stahnma
23. Recovery Time decreases with increased team size (up to 200) 23 @stahnma
24. Longest Recovery Times are from teams of one. 24 @stahnma
25. Performance is better with more than one contributor as shown by multiple indicators 25 @stahnma
26. Software is collaborative 26 @stahnma
27. 27 Is “Don’t Deploy on Friday” a real thing?
28. Short Version: Not really. Not for people using CI/CD. 28 @stahnma
29. 29 Branch Information
30. Teams are innovating and experimenting on feature branches 30 @stahnma
31. Success Rate on default branch higher than on non-default branches 31 @stahnma
32. Success Rate at 50p is 80% for default and 58% for non-default branches 32 @stahnma
33. Duration on default branches are faster at every percentile. 33 @stahnma
34. Recovery Time is lower on default branch at every percentile. 34 @stahnma
35. 35 How has the global pandemic impacted team performance?
36. 36 Throughput @stahnma
37. 37 Duration
38. 38 Success Rate @stahnma
39. 39 Recovery Time @stahnma
40. Hypothesis: Fewer distractions* working at home 40 *For some values of distraction. @stahnma
41. 41 Final Thoughts
42. When mapped against survey surveying data, CI users at 50p show up between medium and high performers at an org level (vs project level). 42 @stahnma
43. 43 @stahnma
44. 44 @stahnma
45. If you are average at using a CI platform, you’ll be right on the line between medium and high performer. 45 @stahnma
46. Our most frequent CI users have better outcomes on our four critical metrics 46 @stahnma
47. More collaborators means better outcomes 47 @stahnma
48. 48 We’re hiring. circleci.com/careers
49. 49 Thank you Michael Stahnke @stahnma Special thanks to Ron Powell and Melissa Santos who gathered this data and were able to answer my questions about it.
50. 50 Percentile 2020 Value 2019 Value 5p 0.03 0.03 50p 0.70 0.80 90p 16.03 13.00 95p 32.125 25.47 Mean 8.22 5.76 Throughput @stahnma
51. 51 Percentile 2020 Value 2019 Value 5p 12 sec 10 sec 50p 3.96 min 3.38 min 90p 21.35 min 19.18 min 95p 34.01 min 31.73 min Mean 24.6 min 26.76 min Duration delta in a year @stahnma
52. 52 Percentile 2020 Value 2019 Value 50p 61% 60% 75p 89% 86% 85p 100% 98% Success Rate @stahnma
53. 53 Percentile 2020 Value 2019 Value 5p 2.06 min 2.83 min 50p 55.11 min 52.5 min 90p 39 hours 47 hours 95p 3.4 days 3.93 days Mean 14.85 hours 16.61 hours Recovery Time @stahnma