Is DevOps Success a Potekmin Village?

Successful devops transformations fill up conference tracks. Where are the ones that are struggling? When speaking at a conference, folks often omit the everyday setbacks, the minor failures, the obstacles they couldn’t remove due to management or organization structure. Those get glossed over in the effort to show the success of the transformation. I want to dig into a transformation that is still underway, from, perhaps, an unlikely source. I work at Puppet, Inc. I run our SRE and platform engineering groups. My company is an authority on devops practices and tooling. However, the way we work isn’t perfect. We’ve had some failures, we’ve had some setbacks, we’ve had “duh” moments. We’re still not where we want to be. I’ll cover some lessons learned, some problems we’re still facing without obvious solutions, and how we’ll take progress over perfection any day of the week.

While we’re shipping a well-known tool in the devops space, this will be a tool agnostic talk. I’ll spend time on organizational structure and incentives, lead time to changes, change failures, deployment conundrums -- some we’ve worked around and some we still face today.

Transcript

1. Is DevOps success a potemkin village? Michael Stahnke Director of Engineering @stahnma
2. “any construction (literal or figurative) built solely to deceive others into thinking that a situation is better than it really is. “ @stahnma
3. @stahnma
4. @stahnma
5. @stahnma
6. Michael Stahnke Puppet, Inc Director of Engineering @stahnma
7. @stahnma
8. @stahnma
9. Principles @stahnma
10. Devops : the way we work @stahnma
11. Working together @stahnma
12. Devops != CI/CD @stahnma
13. “best” Practices @stahnma
14. Worst Practices @stahnma
15. Things you probably shouldn’t do. @stahnma
16. Anti-patterns @stahnma
18. We Ship @stahnma
19. On Premises @stahnma
20. The Update Problem @stahnma
21. The cost of being wrong is high @stahnma
22. @stahnma
23. CULTURE @stahnma
24. @stahnma
25. https://www.flickr.com/photos/openscience/10813661054 @stahnma
26. Hypothesis: This build artifact is shippable @stahnma
27. @stahnma
28. @stahnma
29. @stahnma Classic Layout Dev Ops
30. @stahnma Classic Layout Dev Ops Everybody here thinks they’re a better SysAdmin than you are.
31. @stahnma Classic Layout Dev Ops Back Office Finance/ Sales Marketing
32. @stahnma Not a recommendation Dev Ops Also Ops
33. @stahnma Hypocrisy Dev Ops Back Office Finance/ Sales Marketing
34. @stahnma A DevOps team Dev Ops DevOps
35. @stahnma A DevOps team Dev Ops DevOps SRE
36. @stahnma 6 YEARS later Dev IT Back Office Finance/ Sales Marketing Help Desk Data Center SRE
37. @stahnma A Cross-functional team (in dev)
38. Team Sizes too large @stahnma
39. Multiple Walls
40. We can waterfall in 2 weeks. @stahnma
41. Automation @stahnma
42. Over the wall mentality @stahnma
43. Quality Shifted Right @stahnma
44. Of course CI is red @stahnma
45. Tragedy of the commons for integration @stahnma
46. In Failure We Trust @stahnma
47. Automated a bad process @stahnma
48. https://www.flickr.com/photos/pillarofautumn/13770741955
49. @stahnma
50. @stahnma
51. Jenkins Special Sauce @stahnma
52. A sauce that exits 0 @stahnma
53. Some things were needed too quickly for automation @stahnma
54. What’s in this build? What changed? @stahnma
55. Code Freezes @stahnma
56. Lean @stahnma
57. @stahnma
58. Partially Complete features @stahnma
59. Not always shippable @stahnma
60. AnitPattern: Too much system load @stahnma
61. Measurements @stahnma
62. Unable to respond to feedback, so why try to get any? @stahnma
63. If it moves, capture it @stahnma
64. If it moves, capture it….unless it’s difficult. @stahnma
65. Do nearly nothing with all data @stahnma
66. Sharing @stahnma
67. Complaining is sharing…right? @stahnma
68. Celebrate and share successes @stahnma
69. Ship some great software @stahnma
70. Whole Conferences. The stuff you expect. @stahnma
71. Some Solutions @stahnma
72. If everything starts on the right, you can only shift left. @stahnma
73. Combined QA with Dev @stahnma
74. Deploy releases per sprint (minimally) @stahnma
75. CI has meaning again @stahnma
76. @stahnma
77. Throttle the system @stahnma
78. Feature Toggle Development @stahnma
79. A culture change: This is shippable. @stahnma
80. I know this change-set is good. @stahnma
81. Shipping is learning. @stahnma
82. Modern Issues @stahnma
83. https://www.flickr.com/photos/openscience/10813661054
84. What’s the cost of being wrong? @stahnma
85. AnitPattern: Perf afterwards @stahnma
86. AnitPattern: Too many tests @stahnma
87. AnitPattern: Not the right coverage @stahnma
88. Forming teams around backlogs vs backlogs to teams @stahnma
89. Relearn the same mistakes, but this time with a new language/ toolchain/technology @stahnma
90. Final Thoughts @stahnma
91. Nobody is the best. @stahnma
92. Reduce the number of variables @stahnma
93. Ruthless pursuit of improvements @stahnma
94. This isn’t easy. @stahnma
95. Progress over Perfection @stahnma
96. Organizational Inertia is a thing @stahnma
99. You won’t be done. There is no done. Done is a myth. @stahnma
100. Get Better at Getting Better
101. cheers. @stahnma