Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Infra as Code: The Organizational Bottleneck an...

Evan Gilman
September 29, 2016

Infra as Code: The Organizational Bottleneck and the Shared Service Anti-Pattern

As your company's Engineering team grows, and the rate of infrastructure evolution increases, it becomes increasingly apparent that it is not only your Ops team that needs to manipulate the infrastructure codebase - it's everyone! So tools are built, skills are taught, and before you know it, the infrastructure codebase is the most contentious in the whole organization. This talk chronicles PagerDuty's journey from Ops-driven evolution to distributed evolution, the pains involved and the hard lessons learned.

Evan Gilman

September 29, 2016
Tweet

More Decks by Evan Gilman

Other Decks in Technology

Transcript

  1. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Infra as Code at PagerDuty Chef Evolution at PagerDuty Human Implication of Shared Services What We’re Trying and How to Avoid Agenda
  2. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Chef at PagerDuty
  3. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks Chef at PagerDuty
  4. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks Chef at PagerDuty
  5. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo Chef at PagerDuty
  6. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Per-App Cookbooks • Many Common Cookbooks • Mono-Repo • No Versioning Chef at PagerDuty
  7. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search PD-Specific Tooling
  8. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls PD-Specific Tooling
  9. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad
  10. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN PD-Specific Tooling • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource
  11. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Lots of Chef Search • Lots of External Calls • Linear Scaling == Bad • Chef as Shared Resource • APIs as Shared Resources PD-Specific Tooling
  12. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges Optimization
  13. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Optimization • Instrument Everything • Concerted Effort • Some Specialize • Convention Emerges
  14. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Continued Growth
  15. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! Continued Growth
  16. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! Continued Growth
  17. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice Continued Growth
  18. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth Continued Growth
  19. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
  20. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN • Copy/Paste! • Refactors! • Services rarely identical in practice • Limited Ops bandwidth • Service creation time skyrockets Continued Growth
  21. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs
  22. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful
  23. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Solution? Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart
  24. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Dev needs != Ops needs Chef interface complex and powerful Like an ATV instead of a Golf Cart Focus on Dev experience Solution?
  25. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
  26. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused
  27. 9/25/16 @evan2645 INFRA AS CODE: THE ORGANIZATIONAL BOTTLENECK AND THE

    SHARED SERVICE ANTI-PATTERN Always Know Your Customer Focus on the Infrastructure Interface/API Don’t Fight Conway’s Law Be Product-Focused