What I Learned From 5 Years Sciencing the Crap Out Of Devops

@jezhumble | devopsdays dallas 2018 what i learned from 5
years sciencing the crap out of devops

@jezhumble get it while it’s hot! https://cloudplatformonline.com/2018-state-of-devops.html or http://bit.ly/2018-devops-report

@jezhumble things about technical practices how to make your data
suck less: * writing good survey questions * making sure the survey questions are good - with SCIENCE * (these methods also apply to your system and log data) what we found… that we did (AND didn’t) expect things about management agenda

Dr. Nicole Forsgren Lead investigator, PhD CEO and Chief Scientist,
DORA Diet Coke lover* * Nicole wrote this slide

@jezhumble Not all data is created equal who thinks surveys
suck? who LOVES the data from their logs?

@jezhumble what is a latent construct?

@jezhumble PSYCHOMETRICS We use to make our data look good*
* or give us a reasonable assurance that it’s telling us what we think it’s telling us (& some of this can also apply to your log data)

@jezhumble psychometrics includes: Construct creation (manual) • When possible: use
previously validated constructs • Based on deﬁnitions and theory, carefully and precisely worded, card sorting task, pilot tested Construct evaluation (statistics) • Establishing validity: discriminant and convergent • Establishing reliability

@jezhumble psychometrics writing example: culture Does it matter to our
study? • More than just intuition? What KIND of culture? • National identity and norms • Adaptive culture • Value learning (2014 study) • Value information ﬂow and trust (2014-2018 studies: Westrum)

@jezhumble Westrum, “A Typology of Organizational Cultures” | http://bmj.co/1BRGh5q how
organizations process information try writing items yourself! Use strong statements with clear language

@jezhumble westrum culture items • On my team, information is
actively sought. • On my team, failures are learning opportunities, and messengers of them are not punished. • On my team, responsibilities are shared. • On my team, cross-functional collaboration is encouraged and rewarded. • On my team, failure causes inquiry. • On my team, new ideas are welcomed. found to be valid and reliable Predictive of IT and organizational performance

@jezhumble psychometrics analysis example Notification of failure At my organization:
• We are primarily notified of failures by reports from customers. • We are primarily notified of failures by the NOC. • We get failure alerts from logging and monitoring systems. • We monitor system health based on threshold   warnings (ex. CPU exceeds 100%). • We monitor system health based on rate-of-change   warnings (ex. CPU usage has increased by 25% over the last 10 minutes). Original in 2014, but there was a surprise, can you spot it?

@jezhumble psychometrics analysis example Notification of failure At my organization:
• We are primarily notified of failures by reports from customers. • We are primarily notified of failures by the NOC. • We get failure alerts from logging and monitoring systems. • We monitor system health based on threshold   warnings (ex. CPU exceeds 100%). • We monitor system health based on rate-of-change   warnings (ex. CPU usage has increased by 25% over the last 10 minutes). notification from FAR notification from NEAR

@jezhumble more data tests! Plus, we test to make sure
the survey doesn’t have other problems. • Common method variance (CMV) (aka CMB for Bias) • Early vs. late responders • Survey drop-oﬀ rates and bias

@jezhumble a note about analysis methods One of three conditions
must be met: • Randomized, experimental design (no, this is non-experimental) • Longitudinal (no, this is cross-sectional) • Theory-based design When this condition was not met, only correlations were tested and reported .

@jezhumble OK now we can look at the data and
how they relate to each other

@jezhumble software delivery as a competitive advantage “Firms with high-performing
IT organizations were twice as likely to exceed their proﬁtability, market share and productivity goals.” http://bit.ly/2014-devops-report

software delivery as a competitive advantage high performers were more
than twice as likely to achieve or exceed the following objectives: • Quantity of products or services • Operating eﬃciency • Customer satisfaction • Quality of products or services provided • Achieving organizational and mission goals • Measures that demonstrate to external parties whether or not the organization is achieving intended results http://bit.ly/2017-devops-report

@jezhumble time to restore service lead time for changes (version
control to production) deploy frequency change fail rate software delivery performance http://bit.ly/2014-devops-report

@jezhumble 2018 performance benchmarks http://bit.ly/2018-devops-report

elite performers http://bit.ly/2018-devops-report Data shows a new 4th high performance
group: elite performers Proportion of high performers has grown YoY, but the bar for excellence remains high Elite performers are still able to optimize for throughput and stability

availability http://bit.ly/2018-devops-report Ability for teams to ensure their product or
service can be accessed by end users Software delivery + availability = SDO performance Elite performers are 3.5X more likely to have strong availability practices

capabilities that drive high performance Accelerate: The Science of Lean
Software and DevOps, Forsgren, Humble and Kim 2018

technical practices http://bit.ly/2018-devops-report

@jezhumble key ﬁnding: doing cloud right http://bit.ly/2018-devops-report | NIST SP
800-145 AGREED OR STRONGLY AGREED On-demand self-service Broad network access Resource Pooling Rapid elasticity Measured service Only 22% of teams are doing cloud right! Teams that use these essentials characteristics are 23X more likely to be elite performers

@jezhumble key ﬁnding: architectural outcomes can my team… …make large-scale
changes to the design of its system without the permission of somebody outside the team or depending on other teams? …complete its work without needing ﬁne-grained communication and coordination with people outside the team? …deploy and release its product or service on demand, independently of other services the product or service depends upon? …do most of its testing on demand, without requiring an integrated test environment? …perform deployments during normal business hours with negligible downtime? http://bit.ly/2017-devops-report | https://devops-research.com/research.html | DORA / Puppet

@jezhumble some surprises

@jezhumble which of these measure effective test practices? • Developers
primarily create & maintain acceptance tests • QA primarily create & maintain acceptance tests • Primarily created & maintained by outsourced party • When automated tests pass, I’m confident the software is releasable • Test failures are likely to indicate a real defect • It’s easy for developers to fix acceptance tests • Developers share a common pool of test servers to reproduce failures • Developers create on demand test environments • Developers use their own dev environments to reproduce failures

@jezhumble continuous testing previous practices plus… • continuously reviewing and
improving test suites to better ﬁnd defects and keep complexity and cost under control • allowing testers to work alongside developers throughout the software development and delivery process • performing manual test activities such as exploratory testing, usability testing, and acceptance testing throughout the delivery process • having developers practice test-driven development by writing unit tests before writing production code for all changes to the codebase • being able to get feedback from automated tests in less than ten minutes both on local workstations and from a CI server http://bit.ly/2018-devops-report | https://devops-research.com/research.html | DORA / Puppet

@jezhumble monitoring and observability MONITORING is tooling or a technical
solution that allows teams to watch and understand the state of their systems and is based on gathering predeﬁned sets of metrics or logs. OBSERVABILITY is tooling or a technical solution that allows teams to actively debug their system and explore properties and patterns they have not deﬁned in advance. Teams with a comprehensive monitoring and observability solution were 1.3 times more likely to be an elite performer. Having a monitoring and observability solution positively contributed to SDO performance. Fun stats fact: monitoring and observability load together.

@jezhumble we all know managing work in process (WIP) is
important, right? correlation between WIP and ITPerf is almost zero what’s going on? now for management stuﬀ

@jezhumble lean management

@jezhumble lean product management

@jezhumble software delivery matters (but you have to do it
right) even if you think it’s obvious, test with data • if the results don’t surprise you, you’re doing it wrong • if you don’t also conﬁrm some things you expected, you’re doing it wrong we can have it all, or at least throughput and stability devops culture and practices have a measurable impact on software delivery performance conclusions

thank you! © 2016-18 DevOps Research and Assessment LLC https://continuous-delivery.com/
To receive the following: • A copy of this presentation • The link to the 2018 Accelerate State of DevOps Report (and previous years) • A 100 page excerpt from Lean Enterprise • Excerpts from the DevOps Handbook and Accelerate • 30% oﬀ my video workshop: creating high performance organizations • A 20m preview of my Continuous Delivery video workshop • Discount code for CD video + interviews with Eric Ries & more Just pick up your phone and send an email To: [email protected] Subject: devops

What I Learned From 5 Years Sciencing the Crap ...

What I Learned From 5 Years Sciencing the Crap Out Of Devops

More Decks by Jez Humble

Other Decks in Technology

Featured

Transcript