Devops Behavioral Economics: Displayed behavior as compared to reported behavio

What can you learn about devops and software delivery practices by looking at data from a platform with more than 300,000 developers, 25,000 organizations and 25+ million builds per month? As I recently joined CircleCI, I was interested in this new data set from a large SaaS developer platform and the kinds of questions that it could answer: What trends and patterns pop out from the data? Are they different than what is seen through surveys where responders opt-in to participating as compared to being aggregated through platform usage?

I wanted to use this data set in a behavioral economics methodology to see what reported behaviors are when compared actual behaviors across a large data set. In this talk, I’ll cover a view into anonymized team data from millions of builds to share insights, behaviors, and metrics that help teams build better software faster, and look into characteristics of success that we can measure through data. Finally, I’ll cover what we can infer from team behavior, callout a few tools, start a language war, and provide take-aways for you to benchmark with your own software delivery teams.

Transcript

1. @stahnma 2018
2. @stahnma
3. @stahnma Hard Hitting Stats • The most common length of a repository name is 17.6 characters • The most common project name is “api”, followed by “terraform”, “infrastructure”, and “web” • After environmental names (e.g dev, test, stage) the most common branch name is “update-readme” • 0.01% of workflows have English curse words in the name of the branch
4. @stahnma
5. @stahnma
6. @stahnma MTTR = Time from red to green Delivery lead time = time of workflow? Deployment frequency = how often do you kick off workflows? Change failure rate = workflow failure rate? Mapping Metrics
7. @stahnma Fastest Recovery < 1 second MTTR
8. @stahnma Fastest Recovery < 1 second MTTR
9. @stahnma Put stuff here
10. @stahnma Lead Time Some people would cry for that 50p length, question is would it be tears of sorrow or joy Min 50p 95p Max 2.1s 3m27s 28m45s 3.3 days
11. @stahnma Lead Time Some people would cry for that 50p length, question is would it be tears of sorrow or joy Min 50p 95p Max 2.1s 3m27s 28m45s 3.3 days
12. @stahnma 25 Number of runs per day per project Number of projects building this many
 times per day Deployment Frequency
13. @stahnma 50p 95p 99p Max workflows per day per project 3 29 74 2488 workflows per org per day 6 85 250 3036 workflows per project on master branch per day 1 13 39 2443 Deployment Frequency
14. @stahnma 50p 95p 99p Max workflows per day per project 3 29 74 2488 workflows per org per day 6 85 250 3036 workflows per project on master branch per day 1 13 39 2443 Deployment Frequency
15. @stahnma Change Failure Rate Normal failure rate of roughly ~20%
16. @stahnma 50p of approvals happen in 3.53 minutes 3% for workflows with approval jobs 20% for all workflows Manual Approval Jobs
17. @stahnma Manual Approval Jobs 50p Length (wthout an approval) : 3m27s 50p Length with Approvals: 14m38s Some workflows have more than 1 approval job 3.5% of jobs using manual approvals
18. @stahnma Language Fail rate on master Median branches Median users PHP 7.0% 5 2 JavaScript 9.5% 5 2 Python 10.6% 5 2 Ruby 10.9% 9 3 Go 12.4% 4 2 Java 13.1% 4 2
19. @stahnma Language Fail rate on master Median branches Median users PHP 7.0% 5 2 JavaScript 9.5% 5 2 Python 10.6% 5 2 Ruby 10.9% 9 3 Go 12.4% 4 2 Java 13.1% 4 2
20. @stahnma Open Space? Thank you.