Agile Manchester: What we learned whilst handing control of deployments to developers at Auto Trader

Slightly tweaked version of this presentation for Agile Manchester in May 2016.

Transcript

1. WHAT WE LEARNED WHILST HANDING CONTROL OF DEPLOYMENTS TO DEVELOPERS AT AUTO TRADER MARK CROSSFIELD, MAY 2016
2. Tech Lead in Operations @mrmanc / mark@markcrossfield.co.uk http://markcrossfield.co.uk http://www.devopsmanchester.co.uk/ Fascination for underground features: http://hidden-manchester.org.uk/ MARK CROSSFIELD
3. hidden-manchester.org.uk/
4. Hiring! http://careers.autotrader.co.uk/ © OpenStreetMap contributors
5. http://technw.uk/groups @technwuk
6. About Auto Trader Some deployment history Automated Deployment v1: Pipelines! What did we learn? What is next? STRUCTURE
7. DEVELOPER AT HEART but I care deeply about Operations
8. ‘GUYS’ ˁ
9. TWO YEARS IN OPS Eight years at Auto Trader
11. https://twitter.com/jessitron/status/729028421930655745
12. http://blog.jessitron.com/2016/05/tradeoffs-in-coordination-among-teams.html “Shared tools are great when they’re maintained as internal services, by teams with internal customers” —Jessica Kerr
13. Devops Team Delivery Delivery Delivery ‘Devops Team’ Sweet sweet prod
14. Service Team Delivery Delivery Delivery Sweet sweet prod ‘Devops Team’
15. LOTS OF PEOPLE have contributed to this over the course of years Although I’m telling the story…
17. JOURNEY This was, is and continues to be a
18. CLASSIFIED ADVERTISING We don’t sell cars… Auto Trader’s business is
19. Website launched in 1996
20. FINAL PRINT EDITION June 2013
21. AUTO TRADER PLC Investor owned Today…
22. 4 MILLION PAGE VIEWS / HOUR At peak users generate data taken from Google Analytics
23. Multi faceted & geographic search over ~480k vehicles Large volume of traffic Large number of applications behind the scenes Lack of vehicle sale visibility masks business metrics CORE CHALLENGES
24. 2 x data centre suites each containing: F5 BIG-IP DNS Global Traffic Management F5 Local Traffic Management ~2,400 virtual servers using VMware ~200 HP Blades NetApp Storage INFRASTRUCTURE
26. Java: Spring Boot / Dropwizard Solr, Oracle, Mongo, MySQL, SQL Server, Vertica Jetty / JBoss Apache CentOS / OEL STACK
27. Where We Are
28. Continuous Delivery & Continuous Learning Tens of experiments per day WHERE ARE WE GOING?
29. 2007: Copied .war files 2008: Standardised script built RPMs, Perl deployment script, hired team of release engineers, VMware 2009: Redeveloped autotrader.co.uk built own RPM, introduced Cruise / Go CD 2010: Another project builds own RPM, release engineers absorbed 2011: Further projects build own RPM 2013: New CEO DEPLOYMENT HISTORY
30. bit.ly/SquadsAndTribes
31. SEPT 2013: DEPLOYMENT AUTOMATION
32. GOAL: TO CREATE BREATHING SPACE
33. SO WHAT DID WE DO?
36. LAYERED RPMS allowed developers to roll out config themselves
39. PIPELINE TEMPLATES & DEPENDENCIES allowed us to compose deployment processes from reusable templates
40. Deployment Pipelines
42. SELF CONTAINED APPS
43. Started to roll out Auto Deploy Formed new team in Ops
45. AUG 2015: DEVELOPERS KICK OFF ALL AUTO DEPLOYMENTS !
47. ! Config in the hands of developers ! Deployments are repeatable ! Deployments are faster ! Less people tied up ! No longer constrained by release slots RECAP
48. AWESOME, SO WHAT WAS PAINFUL?
49. THREE THINGS I WOULD CHANGE
50. (MIS)USING OUR CI TOOL
51. THE DEATH STAR How our pipelines looked before: Dec 2013 http://markcrossfield.co.uk/go-dependency-force-layout/
52. THE DEATH STAR How our pipelines looked during: Oct 2014 http://markcrossfield.co.uk/go-dependency-force-layout/
53. THE DEATH STAR How our pipelines looked by Dec 2015 https://flic.kr/p/qngNWb http://markcrossfield.co.uk/go-dependency-force-layout/ I used to bullseye womp rats in my T-16 back home
54. RPMS FOR EVERYTHING
56. THE DOCUMENTATION
57. Stretching continuous integration environments Over-using complicated packaging format Overly complicated documentation is a smell PAIN RECAP
58. THOSE PITFALLS HELPED US REACH OUR GOAL
59. # Senior Leadership Support # Starting with frequent deployers # Piloting with flexible team # Automating existing processes # Building tools and build plugins # Focussed multi skilled team with CI mindset OTHER THINGS WHICH HELPED
60. SENIOR LEADERSHIP SUPPORT
61. 2015: REFOCUSSED ON PRIVATE CLOUD
62. Decouple devs from infrastructure using a clean API Make deployments easy and visible Make deployments faster Reduce config drift Make infrastructure change visible Flow regular updates through teams’ pipelines Facilitate emergency patches Remove innovation bottlenecks GOALS
63. FASTER DEPLOYMENTS DEPLOYING DEPLOYING! https://xkcd.com/303/
64. FASTER BACK OUT https://commons.wikimedia.org/wiki/File:Panic_button.jpg
65. https://commons.wikimedia.org/wiki/File:Panic_button.jpg DEPLOYMENT? YAWN. https://www.flickr.com/photos/marinadelcastell/14754852591/
66. EXISTING CLOUDSTACK & SCALR ENVIRONMENT
67. DEPLOYMENT SERVICE PROVIDES DECOUPLED API
68. AUTOMATED TESTING OF INFRASTRUCTURE CODE
69. INFRASTRUCTURE CODE PACKAGED
71. SAME SPECIFICATION IN EACH ENVIRONMENT
72. PROD GETS PATCHED Because infrastructure flows into app pipelines
73. ANY CHEF RECIPES General enough to manage everything virtual Java as a first class citizen… but
74. EARLY DAYS
75. CLOUD READINESS
78. Get leadership buy in Dedicate people with the right skills and attitude Involve your customers & shout about progress Focus on the MVP Look for shortcuts Reorganise around products Keep It Simple, Stupid CONCLUSIONS
79. Continuous Delivery
 Reliable Software Releases through Build, Test, and Deployment Automation by Jez Humble and David Farley http://www.amazon.co.uk/dp/0321601912 A manifesto for applications running on Heroku, and applicable as a set of best practices for cloud native applications with some great wisdom http://12factor.net/ Berkshelf: Chef Cookbook packaging (part of Chef DK) http://berkshelf.com/ https://www.go.cd/ https://downloads.chef.io/chef-dk/ Scaling Agile @ Spotify http://bit.ly/SquadsAndTribes @mrmanc / http://bit.ly/markatagilemanc