Deploy, Collaborate and Listen

At Etsy we are continuously deploying software with about 50 deploys per day on average. We have about 150 engineers that work together to add features, improve the site, solve problems, and figure out outages. We are constantly working on improving ways to collaborate and have the time to invest into research and new projects instead of fire fighting. However this hasn't been like this from the beginning. In the Dark Ages of Etsy we had a vastly different software architecture, a myriad of silos with a less than ideal amount of communication and a not so stable site. So what happened? How did we fix this?

We will briefly go over how we changed the architecture and culture to make site operations and stability better. After that we will go into detail about what it meant for us to maintain that culture of collaboration and trust and things we learned from that. And finally take a look at the state of things in the present and give a comprehensive picture of how we arrived here and where to go from there.

Transcript

1. DEPLOY COLLABORATE & LISTEN Daniel Schauenberg d@etsy.com @mrtazz
2. ETSY
3. THE MONOLITH
4. LAMP
6. HOW COMFORTABLE ARE YOU DEPLOYING A CHANGE RIGHT NOW
7. MTTR TRUMPS MTBF
8. IF THIS IS YOUR FIRST DAY AT ETSY YOU DEPLOY THE SITE
10. THE DARK PAST
11. ❗
12. - HINDSIGHT IS 20/20 - THERE WOULD BE NO ETSY - I WASN'T AROUND FOR THIS (the grain of salt disclaimer)
13. DARK LESS FUN
14. ARCHITECTURE OVERVIEW - UBUNTU - POSTGRESQL - LIGHTTPD - PHP/PYTHON
15. SINGLE BIG DATABASE
16. BUSINESS LOGIC IN STORED PROCEDURES
18. SILOS
19. DEV ! DBA ! OPS
20. SPROUTER
21. A SOFTWARE MANIFESTATION OF SILOS
22. SITE UPTIME WASN'T GOOD
23. MORE UPTIME > LESS UPTIME
25. Trust THE PEOPLE
27. SPROUTER
28. NO MORE MAINFRAME MASTER
29. HORIZONTAL SCALING (A SINGLE BOX ONLY SCALES SO FAR)
30. MASTER-MASTER REPLICATED MYSQL SHARDS
31. FLICKR DNA
32. DEPLOYINATOR
33. DEPLOY != RELEASE
34. CONFIG FLAGS
35. if Feature::isEnabled($feature) { // new hawtness } else { // nothing to see here }
36. METRICS!
38. STATSD "HOW DO YOU KNOW THIS WORKS IN PRODUCTION?"
39. CHEF
40. <3 CHEF
41. DEVELOPER VMS - KVM - EVERY ENGINEER HAS ONE - FULLY CHEF’D WITH THE ETSY STACK - DIFFERENT SIZES AND CHEF ROLES
42. Try
43. BLAMELESS POST MORTEMS
46. OVERHAULS & ITERATIONS
47. CULTURE & TOOLS
48. HUMANS ARE AWESOME
49. NOBODY COMES TO WORK TO DO A BAD JOB
50. Trust YOUR CO- WORKERS
51. THERE IS A LOT OF KNOWLEDGE IN YOUR ENGINEERING TEAM
52. DEPLOY (AS OFTEN AS IT MAKES SENSE)
53. COLLABORATE (EVEN IF YOU THINK YOU DON'T HAVE TO)
55. LISTEN (TO PROBLEMS AND EXPERIENCES OF YOUR COWORKERS)
56. CODEASCRAFT.COM ETSY.COM/CODEASCRAFT/TALKS ETSY.COM/CAREERS
57. THANK YOU!
58. DEPLOY COLLABORATE & LISTEN Daniel Schauenberg d@etsy.com @mrtazz