The Future of Ops

@tyler_treat Tyler Treat • DevOpsDays Des Moines • 4/13/18 The
Future of Ops

@tyler_treat Welcome to the  world of tomorrow!

@tyler_treat @tyler_treat

@tyler_treat Tyler Treat Managing Partner @ Real Kinetic Former infrastructure
engineering manager @ Workiva bravenewgeek.com

@tyler_treat Data Center

@tyler_treat Data Center Compute Network Storage

@tyler_treat Data Center Compute Network Storage App Servers Security Backups/DR
Monitoring

@tyler_treat Data Center Compute Network Storage Help Desk Procurement Compliance
App Servers Security Backups/DR Monitoring

App Servers Security Backups/DR Monitoring App App App App App App App

App Servers Security Backups/DR Monitoring

App Servers Security Backups/DR Monitoring App App App App App App App Ops

App Servers Security Backups/DR Monitoring App App App App App App App DevOps

@tyler_treat App App App App App App App NoOps

@tyler_treat App App App App App App App Infrastructure Automation
Deployment Automation Conﬁguration Management Log Management Monitoring NewOps

@tyler_treat

@tyler_treat DevOps is a journey, not a destination.

@tyler_treat Manual Provisioning Self-Service The DevOps Scale of Automation

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup

@tyler_treat Scaling DevOps

@tyler_treat @tyler_treat Why do silos form?

@tyler_treat Many companies start with a “DevOps” approach.

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup DevOps by
Necessity  Devs push to production, unstable, high-risk, minimal cost control

@tyler_treat As the product scales, we specialize. @tyler_treat

@tyler_treat As the business scales,  we add safety checks.

@tyler_treat Developers write code.

@tyler_treat Ops people run it.

@tyler_treat QA gets blamed for defects.

@tyler_treat Security blocks everything.

@tyler_treat And management wonders why nothing gets shipped.

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup Ops as
Gatekeepers  Stable, cost-controlled, risk-averse,  delivery and innovation bottleneck 

@tyler_treat Specialization is good!

@tyler_treat Misalignment is not good.

@tyler_treat How do we scale specialization?

@tyler_treat Cross-functional teams? @tyler_treat

@tyler_treat DevOps encourages cooperation!

@tyler_treat Just add an ops engineer to each team.

@tyler_treat And maybe a reliability engineer.

@tyler_treat Maybe a few extra for on-call backup.

@tyler_treat And of course we need a QA engineer too.

@tyler_treat Done!

@tyler_treat Also, $$$

@tyler_treat How do we scale specialization?

@tyler_treat Vision and Product

@tyler_treat Vision: a mental image of what the future could
be like. @tyler_treat

@tyler_treat Vision enables independent decision making and alignment.

@tyler_treat But vision without execution is just hallucination…

@tyler_treat Products are how we scale execution.

@tyler_treat

@tyler_treat The evolution of QA Test-focused Tools-focused

@tyler_treat The evolution of QA QA SDET “Combined” Engineering

@tyler_treat Production CD Pipeline CI

@tyler_treat QA teams are shrinking, but what’s growing are the
teams building the tools.

@tyler_treat The same is becoming true of Ops.

@tyler_treat build/release/deploy conﬁguration management infrastructure automation logging & instrumentation monitoring

@tyler_treat By productizing our infrastructure, we scaled.

@tyler_treat We controlled costs.

@tyler_treat We reduced risk.

@tyler_treat We accelerated development.

@tyler_treat We delivered value to customers faster…

@tyler_treat from 3 - 4 releases per year to multiple
releases per day.

@tyler_treat Rethinking Ops

@tyler_treat

App Servers Security Backups/DR Monitoring App App App App App App App Wake me up if anything goes wrong here. Ops as Masters of Production

App Servers Security Backups/DR Monitoring App App App App App App App Jim Bob’s Frobulator service is out of memory. Ops as Masters of Production

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup PaaS  Stable,
cost-controlled, risk-averse,  delivery enabler, innovation bottleneck

@tyler_treat

@tyler_treat Enable developers to self-service through tooling and automation and
empower them to deploy and operate their services… @tyler_treat The Vision

@tyler_treat “Here’s a CloudFormation template and access to production…”

@tyler_treat Manual Provisioning Self-Service Large Enterprise Small Startup IaaS  Devs
provision infrastructure as code, free-for-all, cost explosion, high-risk,  delivery and innovation enabler

empower them to deploy and operate their services… @tyler_treat The Vision

empower them to deploy and operate their services…with minimal Ops intervention. @tyler_treat The Vision

@tyler_treat

Deployment Automation Conﬁguration Management Log Management Monitoring Ops as Product Team

Deployment Automation Conﬁguration Management Log Management Monitoring Products Ops as Product Team

@tyler_treat Pain-Driven Development: making locally optimal decisions to minimize pain.

@tyler_treat Silos promote pain displacement. Product Development QA Ops

@tyler_treat Silos promote pain displacement. Product Development QA Ops pain
of running software pain of testing software pain of building software

@tyler_treat Misaligned incentives!

@tyler_treat How do you expect devs to care about quality
if they’re not on the hook?

@tyler_treat How do you expect devs to care about operability
if they’re not on the hook?

@tyler_treat Devs won’t build truly reliable systems until they are
on-call for them.

@tyler_treat BUT!

@tyler_treat Responsibility requires empowerment.

@tyler_treat You can’t ask someone to care about something and
ﬁx it without also giving them the power to do so.

@tyler_treat Most Ops teams simply haven’t done enough to empower
and offload responsibility onto dev teams.

@tyler_treat Products enable ownership.

Deployment Automation Conﬁguration Management Log Management Monitoring Products Ops as Product Team

Deployment Automation Conﬁguration Management Log Management Monitoring Products The Frobulator service is out of memory… Since you are the Frobulator expert, here are these tools to help you diagnose and resolve the problem autonomously. Ops as Product Team

@tyler_treat Products maintain control through enablement.

@tyler_treat Enable teams to follow best practices.

@tyler_treat Best practices for builds.

@tyler_treat Best practices for testing.

@tyler_treat Best practices for deploys.

@tyler_treat Best practices for support.

@tyler_treat Best practices for compliance.

@tyler_treat Encode compliance and SDLC requirements into tooling and process.

@tyler_treat Snowflakes kill…  Use pain-driven development to your advantage by
creating paths of least resistance.

@tyler_treat Teams must make a case for going off-menu.

@tyler_treat Products in Practice

@tyler_treat Build Release Deploy Operate

@tyler_treat Code Repository Dev Push change to branch Review by
Peers Build QA & Compliance Continuous Integration

@tyler_treat

@tyler_treat • Build plan part of the code, not baked
into build tool  • Dev teams fully control their builds  • Deep integration with GitHub   • Build controls into the process Continuous Integration

@tyler_treat Code Repository Dev Tag branch for release Build/QA Continuous
Delivery Dev Artifact Repository Sign-Off Prod Artifact Repository Deploy

@tyler_treat • Artifact build/tagging/promotion automation  • Container/machine image auditing  •
Machine image and security patch automation  • Streamlining sign-off from different parties Continuous Delivery

@tyler_treat

@tyler_treat • Self-service deploys  • Self-service conﬁguration (with guard rails) 
• Infrastructure provisioning is automated  • No ticket-driven development Continuous Deployment

@tyler_treat • Logging  - Structured logging spec  - Language libs
implementing spec  - Log pipeline (i.e. agent, collector, storage, search)  • Telemetry, tracing, health checks, alerting  • Canary deploys, A/B testing, trafﬁc shadowing, etc. Continuous Operations

@tyler_treat Many off-the-shelf solutions just need “glued” together.

@tyler_treat Most problems are cultural, not technical.

@tyler_treat Technology will not ﬁx your broken culture!

@tyler_treat Solutions need to ﬁt the company, its culture, and
its architecture.

@tyler_treat Get the workflow correct, start manual, then automate.

@tyler_treat Wrapping Up

@tyler_treat Specialization leads to misalignment and broken feedback loops.

@tyler_treat But specialization is an important part of scaling a
business.

@tyler_treat The question is:  how do we specialize?

@tyler_treat The traditional Ops model does not scale.

@tyler_treat DevOps is about tightening feedback loops and building empathy.

@tyler_treat NewOps is about empowering teams and providing autonomy.

@tyler_treat It’s not a replacement for DevOps, it’s an evolution
of it.

@tyler_treat It’s applying a product mindset to the traditional Ops
model.

@tyler_treat Ops teams should be redeﬁning their vision:

@tyler_treat from masters of production to enablers of production.

@tyler_treat Ops capabilities should be embedded within dev teams…

@tyler_treat but they need to be enabled!

@tyler_treat

@tyler_treat NewOps treats Ops like a product team whose product
is infrastructure.

@tyler_treat Creating guard rails, not walls.

@tyler_treat Offloading responsibility helps correct and scale feedback loops.

@tyler_treat Traditional Ops isn’t going away, it’s just getting a
product manager.

@tyler_treat Thanks!  bravenewgeek.com realkinetic.com

The Future of Ops

The Future of Ops

More Decks by Tyler Treat

Other Decks in Technology

Featured

Transcript