How JUST EAT Leveraged AWS to Foster a DevOps Culture (without using the word ‘DevOps’)

A talk I gave at the Agile Vancouver Conference (Much Ado About Agile X) on Oct 26th, 2015 around how JUST EAT was able to use AWS (Amazon Web Services) to transform its organization and systems and foster a culture of trust, responsibility and authority.

The transitions and clips haven't made it to this publication, sorry about that, but you can see the video of the talk at https://youtu.be/8v3EAursPho

Transcript

1. Byron Packwood, Technology Fellow JUST EAT @byronp
2. (WITHOUT USING THE WORD ‘DEVOPS’) Except for twice, in this title slide How JUST EAT Leveraged AWS to Foster a DevOps Culture
3. How JUST EAT Leveraged AWS to Foster a DevOps Culture Address our Challenges, Achieve our Goals and Transform our Culture
5. Here in Vancouver
6. 15 countries 59K restaurants 11M active consumers 2015
7. 2011 2012 2013 2014 61.2 40.2 25.3 13.9 ORDERS PER YEAR (M)
8. The Challenge The Goals The Principles The Outcomes
9. The Challenge The Goals The Principles The Outcomes
10. 2008-1 2008-39 2009-24 2010-9 2010-47 2011-32 2012-17 UK PEAK ORDER RATES
11. SCALE
12. The Challenge The Goals The Principles The Outcomes
13. GO TO THE CLOUD DO MORE AGILE DO DEVOPS
15. GO FASTER BE MORE RESILIENT FOCUS ON CUSTOMERS AND RESTAURANTS
16. Engineering KPIs High Availability Rate of release
17. Engineering KPIs High Availability Rate of release
18. The Challenge The Goals The Principles The Outcomes
19. RESPONSIBILITY different than AUTHORITY
20. RESPONSIBILITY different than AUTHORITY
21. Align with RESPONSIBILITY AUTHORITY
22. No gates
23. Design for Failure
24. How we did it
25. Conway’s Law “organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations” -Melvin Conway, How Do Committees Invent, 1968
26. “Inverse Conway Maneuver” Evolve your organizational structure to promote your desired architecture -Thoughtworks Technology Radar, July 2014
27. Re-organized Small, independent teams Own development and operations of their piece of the platform Created a Platform Services team to enable rapid, confident change
28. You build it , you run it
30. AWS Scale “Every day, AWS adds enough new server capacity to support all of Amazon’s global infrastructure when it was a $7B annual revenue enterprise (in 2004)” -James Hamilton, VP & Distinguished Engineer, Amazon - AWS Innovation at Scale, re:invent 2014 (SPOT301)
31. 0 150 300 450 600 2008 2009 2010 2011 2012 2013 2014 516 280 159 82 61 48 24 AWS Pace of Innovation
32. Resilience to instance failure Resilience to AZ failure
34. But not region failure
35. Autoscaling Elastic Load Balancing Virtual Private Cloud
36. Split up the platform into Cloudformation stacks
37. Cloudformation Create and manage a set of AWS resources Define resources in template using JSON Easy to validate, can inspect Easy to consume and generate Easy to diff and source control Important to keep DRY
38. Cloudformation 165 stacks in production
39. Platform Services Deployment & Environment Monitoring Service Centralized Logging Alerting Service
40. Deployment & Environment Services Tooling to create VPCs, subnets (aka network shell) Platform bootstrapping NAT Securitygroups Stackman (EIP Management)
41. Deployment & Environment Services JustDeploy Service to deploy features to an environment
42. Deployment & Environment Services JustDeploy Service to deploy features to an environment
43. Monitoring Service Using statsd & Graphite ecosystem (Grafana, Tasseo, etc) Receiving about 1.5 datapoints per minute (system via perftap & diamond, AWS and features)
44. Alerting Service Use Seyren (Graphite-based) to alert off what we graph
45. Alerting Service Graphite-based, alert definition is YAML Watchman is command line tool to push alert definition to Alerting
46. Alerting Service Notifications get to engineers via Pagerduty & also to Hipchat rooms for collective visibility
47. Centralized Logging Run ELK as a stack (ElasticSearch, Logstash and Kibana) along with log push agents (nxlog, rysyslog)
48. Centralized Logging We ingest about 250K documents (around 350GB) per day and keep ~3 weeks online and available
49. Equanimity as a Service Early days of operations for many Interest and aptitude (and responsibility and authority) - some were just missing experience
50. Equanimity as a Service Early days of operations for many Interest and aptitude (and responsibility and authority) - some were just missing experience
51. Equanimity as a Service Early days of operations for many Interest and aptitude (and responsibility and authority) - some were just missing experience
52. Platform Features Platform Services Centralized Logging Environment Alerting Backup Deployment Email HAProxy Monitoring NAT PaaSBot SecurityGroups SessionState SQL Server Stackman
53. The Challenge The Goals The Principles The Outcomes
54. Growth (in people)
55. Trust
56. Culture
57. GO FASTER BE MORE RESILIENT FOCUS ON CUSTOMERS AND RESTAURANTS
58. Thanks. @byronp