"Why would anyone do out-of-hours support for free?"

"Why would anyone do out- of-hours support for free?" Sarah
Wells Principal Engineer, Financial Times @sarahjwells

@sarahjwells Hello

@sarahjwells Why this title?

What I learned about devops at the FT

@sarahjwells 1. Why devops? 2. What we did 3. The
hard stuﬀ

@sarahjwells First: the Financial Times

@sarahjwells “… one of the world's leading business news and
information organisations…”

@sarahjwells We need to be able to move fast

@sarahjwells Why did we think devops would help?

@sarahjwells https://puppet.com/resources/white-paper/2016-state-devops-report/

@sarahjwells But nothing comes for free

@sarahjwells Doing devops properly means changing your culture

@sarahjwells What motivated us to change?

@sarahjwells People were frustrated

@sarahjwells Slow

@sarahjwells It took months to get a production server

@sarahjwells It took days to get a release ready

@sarahjwells It took hours to do that release

@sarahjwells And we did those releases once a month

@sarahjwells Processes were manual and error prone

@sarahjwells All our servers were snowﬂake servers

@sarahjwells Release instructions were in a spreadsheet

@sarahjwells Releasing was a box ticking exercise

@sarahjwells Preconditions for change

@sarahjwells A change in how the business thought about technology

@sarahjwells Big, complex new projects

@sarahjwells An in-house example of doing things diﬀerently

hard stuﬀ

@sarahjwells Infrastructure as code

@sarahjwells Developers could provision VMs

@sarahjwells Built-in support for the things that mattered

@sarahjwells Collaboration

@sarahjwells

@sarahjwells Tech ops joined teams

@sarahjwells Groups of people with a common interest

@sarahjwells “You build it, you run it”

@sarahjwells If you aren’t doing this, you aren’t doing devops

@sarahjwells This made developers focus on building for operability

@sarahjwells Support via an “engineering checklist”

@sarahjwells Started by handling operations “in hours”

@sarahjwells “Ops cops” and a dedicated kanban lane

@sarahjwells Providing quick ways to ﬁnd out about and investigate
issues

@sarahjwells Need to share knowledge widely

@sarahjwells Programme ‘curriculums’

hard stuﬀ

@sarahjwells The most difficult thing to do is change your
culture

@sarahjwells Persuading operations to trust developers

@sarahjwells I could provision a VM but not do the
ﬁrst deploy

@sarahjwells Development teams needed to prove we cared about operability

@sarahjwells It helped having second line support on the teams

@sarahjwells Empowering teams, not just talking about it

@sarahjwells Who makes the technology decisions?

@sarahjwells Empowerment also means the freedom NOT to use internal
services

@sarahjwells –http://matt.chadburn.co.uk/notes/teams-as-services.html “the choice to decide who is going to
provide a service is typically around ease of use (for the team), readiness, and self-sufficiency, rather than the being too concerned about the politics of vertical integration across the company”

@sarahjwells Tools and standards are better than platforms

@sarahjwells Out-of-hours support

@sarahjwells Operations teams worry about their roles changing

@sarahjwells Developers worry about having to do a second shift

@sarahjwells There are lots of reasons people may not want
to be on call

@sarahjwells So what have we settled on?

@sarahjwells We still have ﬁrst and second line support on
duty or on call

@sarahjwells It’s not one size ﬁts all

@sarahjwells Small teams vs large teams

@sarahjwells Don’t be afraid to try things out

@sarahjwells Don’t treat every decision like it’s irreversible http://uk.businessinsider.com/jeﬀ-bezos-on-type-1-and-type-2-decisions-2016-4

@sarahjwells So why *would* someone do out-of-hours support for free?

@sarahjwells What diﬀerence has devops made at the FT?

@sarahjwells A new service can be put into production in
minutes not months

@sarahjwells Zero down time deployments are normal

@sarahjwells Minutes from test sign oﬀ to live in production

@sarahjwells My team have released to production over 1400 times
this year

@sarahjwells Our Google AMP integration was built in weeks

hard stuﬀ

@sarahjwells Thank you!

"Why would anyone do out-of-hours support for f...

"Why would anyone do out-of-hours support for free?"

More Decks by Sarah Wells

Other Decks in Programming

Featured

Transcript