Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern

As your company's Engineering team grows, and the rate of infrastructure evolution increases, it becomes increasingly apparent that it is not only your Ops team that needs to manipulate the infrastructure codebase - it's everyone! So tools are built, skills are taught, and before you know it, the infrastructure codebase is the most contentious in the whole organization. This talk chronicles PagerDuty's journey from Ops-driven evolution to distributed evolution, the pains involved and the hard lessons learned.

Transcript

1. 9/25/16 @evan2645 Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern EVAN GILMAN
2. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN About Me
3. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN
4. 9/25/16 @evan2645 This Talk is NOT RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
5. 9/25/16 @evan2645 This Talk IS RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
6. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Infra as Code at PagerDuty Chef Evolution at PagerDuty Human Implication of Shared Services What We’re Trying and How to Avoid Agenda
7. 9/25/16 @evan2645 Chef RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF Nomad + Mesos
8. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Chef at PagerDuty
9. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks Chef at PagerDuty
10. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks Chef at PagerDuty
11. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo Chef at PagerDuty
12. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo • No Versioning Chef at PagerDuty
13. 9/25/16 @evan2645 Through the Ages RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
14. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Baby Chef
15. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Baby Chef
16. 9/25/16 @evan2645 PD-Specific Tooling RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
17. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Lots of Chef Search PD-Specific Tooling
18. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls PD-Specific Tooling
19. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad
20. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource
21. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource • APIs as Shared Resources PD-Specific Tooling
22. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges Optimization
23. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Optimization • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges
24. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Continued Growth
25. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! Continued Growth
26. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! Continued Growth
27. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice Continued Growth
28. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth Continued Growth
29. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
30. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
31. 9/25/16 @evan2645 Chef For All! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
32. 9/25/16 @evan2645 RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF Strong Convention
33. 9/25/16 @evan2645 CONTRIBUTING.md RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
34. 9/25/16 @evan2645 CONTRIBUTING.md RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
35. 9/25/16 @evan2645 But… I just copy/pasted! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
36. 9/25/16 @evan2645 But… It’s just one query… RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
37. 9/25/16 @evan2645 Hey… That’s not in CONTRIBUTING.md! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
38. 9/25/16 @evan2645 C’mon… I just wanna get stuff done RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
39. 9/25/16 @evan2645 Shared Services Considered Harmful RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
40. 9/25/16 @evan2645 Shared Services Considered Harmful RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
41. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Solution?
42. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs
43. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful
44. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart
45. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart Focus on Dev experience Solution?
46. 9/25/16 @evan2645 Data as a Service Definition RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
47. 9/25/16 @evan2645 Artifact + Service Definition RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
48. 9/25/16 @evan2645 CI/CD Pipeline for Workloads RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
49. 9/25/16 @evan2645 CI/CD Pipeline for Workloads RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF (with linters)
50. 9/25/16 @evan2645 Work is Ongoing… RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
51. 9/25/16 @evan2645 Work is Ongoing… RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF (we need help!)
52. 9/25/16 @evan2645 Avoidance? RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
53. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
54. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
55. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
56. 9/25/16 @evan2645 Thank You! RESILIENT INFRASTRUCTURE ORCHESTRATION WITH SERF
57. 9/25/16 @evan2645 Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern EVAN GILMAN