Co-Workers As Customers: Lessons from Airbnb & Etsy

Internal tools aren't known for their beautiful consumer UI, or extreme ease of use, but they are what keeps any company humming.
But this is changing. Airbnb and Etsy are among a handful of companies leading the way in developing not only simple, easy to use tools, but also ones that customers love - and by that I mean colleagues.
These tools make work more efficient and help everyone at the company deliver the best results. Learn from these industry leaders on how and why to build great internal tools and they'll share some of their prized (open source) possessions - Stats D, Deployinator, Chronos.

Transcript

1. CO-workers as customers Lessons from Airbnb & Etsy Harrison Shoff Daniel Schauenberg harry@airbnb.com dschauenberg@etsy.com @hshoff @mrtazz
2. Optional and somewhat awesome footnote text goes here. • StatsD • Deployinator • Morgue • chef-whitelist 2 Daniel Schauenberg Senior Software Engineer at Etsy’s Infrastructure Team
3. New York City
4. Portal Earrings by SVCharms
5. Adventure Time Crochet Dolls by chichirevolver
6. Etsy Stats 6 25M 18M 60M 1.5B members items listed monthly visitors page views per month
7. Optional and somewhat awesome footnote text goes here. Etsy Stack • PHP/Ruby/Node.js/Python/Perl/Go • MySQL/Postgres/Vertica/Hadoop/Solr • MemcacheD/Redis/Gearman • StatsD/Logster/Graphite/Ganglia • Chef/Nagios/IRC 7
9. • Chronos • Photography Tool • JavaScript Style Guide 9 Harrison Shoff Design Engineer at Airbnb
10. San Francisco
11. Krakow, Poland 11
12. Treehouse - Vermont 12
13. 1 sq m House - Berlin, Germany 13
15. As of Dec 19, 2013 Airbnb Stats 15 10M 550k 175 25 6 Million in past year 250,000 past year nights booked properties countries languages
16. Airbnb Stack • Ruby/Java/Node • Hadoop/Hive/Redshift/Mesos/Chronos • StatsD & Graphite (thanks Daniel) • MySQL/Redis/Dyson/S3/Memcached • Chef/Synapse/Nerve/Stemcell 16
17. Why build tools?
18. Enable people
19. Enable people 1x 10x
20. Reproducibility
21. No dependency on providers
22. Nothing says undocumented like, “run this command on this box”
23. Not Invented Here Syndrome be careful about this
24. Getting Started
25. 3 warning signs that you’re missing a tool
26. Outage or Incident
27. It sucks because we have to do X then Y then Z X Y Z
28. Simplify the process X Y Z
29. “There should be a tool for this”
30. 30 Team Structure • 5 Engineers + 1 Manager • 1 Engineer per project • Consulting with developers, ops engineers • Most projects have at least 2 people who occasionally work on it • Each team is also building tools in their field • 12 engineers + 2 Managers • Data Infrastructure • 4 Engineers + 1 Manager • Ops Tools • 8 Engineers + 1 Manager • All building tools to support employees
31. Open Source
32. Why Open Source? Should we give away tools for free? 32
33. You probably owe most of your stack to Open Source
34. Don't reinvent the wheel
35. Item by WillowAndPoppy Give back to the community
36. Item by XStitchMyHeart Contributions and Improvements by the community
37. Open Source The good parts
38. 38 Good Examples Where it worked
39. • Exist in almost any stack in some form • > 680 commits from > 120 contributors • Clients in almost any language • Only custom part is the config file • Pluggable backends • Flexible • Easy to get started • Switched over to the public version early • There is no internal repo 39 StatsD The Poster Child
40. Morgue
41. • Post Mortem Keeper • Timelines • IRC logs • Graphs • Images • Jira Tickets • Added feature API for Open Sourcing • Internal repo with Etsy features • Main version deployed from github.com 41 Morgue
42. http://www.flickr.com/photos/tasselflower/95151097 Kale (Skyline/Oculus)
43. • Anomaly detection engine • Based on Graphite input data • Clearly defined interface/API • Started as a hack week project • Running the Open Source version from the start 43 Kale (Skyline/Oculus)
44. Chronos JavaScript Style Guide How we write JavaScript
46. • Mostly opinions • Helps with code reviews, on boarding • Big team, one codebase => consistency matters • Template for other developer teams to use • +1,400 forks • Only gets better over time 46 JavaScript Style Guide How we write JavaScript
47. Chronos
49. Optional and somewhat awesome footnote text goes here. • Information and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 49 • 10,000+ lines of Bash • 25 sleep statements • Single point of failure Broken Data Pipeline Before Chronos
50. Optional and somewhat awesome footnote text goes here. • Information and stuff • So much information I can’t handle it • Look at all this data and stats and important findings • I can barely contain all this content! 50 • Mesos as a common platform for data infra (spark, shark, hadoop) • Express and visualize dependencies • Retries • < 2,000 lines of Scala • REST api • Web UI for analysts A Better Way Chronos
53. Open Source The bad parts
54. 54 Bad Examples Where we wish we did things differently
55. Deployinator 55
56. • Does all the deploys • One button deploys • Makes it easy to add new stacks • Open Sourced during a conference in 2010 • We wanted to get it out there • Had to rip too much stuff out to make it work internally • Never took the time to get back in sync 56 Deployinator The Rushed One
57. Dashboards 57
58. • Powers all Etsy dashboards • Makes it extremely easy to add new ones • Supports Graphite and Ganglia • Heavily tied to Etsy web code • No clear abstraction • Open sourced a snapshot • Periodic updates 58 Dashboards The Deep Integration
60. • Parses loglines and emits metrics • Support for Graphite and Ganglia • Individual parsers are just Python classes • Open sourced without switching over to the public version • Popular project so it diverged quickly • Internal deploy not set up to sync easily • On-going effort to run the Open Source version 60 Logster The missed opportunity
62. 62 +750 jobs
63. 63 +750 jobs We only tested the UI with 300 jobs…
65. Internal Fork
66. Internal Fork Supporting 2 projects Maintenance priority
68. Back on track
69. Difficulties Why open source?
70. http://www.flickr.com/photos/allthosedetails/7665289260 Extra Work
72. The fork is always the easier route
73. You might be solving a very specific problem
74. People, deal with it!
75. • Enforces some good constraints about testability and separate of concerns • Great for recruiting • New hires are familiar with your tools • Freenode/GitHub does a lot of work for you 75 Experiences Why Open Source?
76. • Have a Freenode IRC channel (#codeascraft) • Have an active GitHub presence • Think about Open Source right from the start • Do not ever run an internal fork • Seriously don’t 76 Tips Why Open Source?
77. • All about enablement • Decrease the time and frustration spent waiting • Engineer happiness++ • Fast feedback (continuous integration, continuous deployment) 77 Tooling Culture Why is a tooling culture important?
78. • Bootcamp with Tools teams • Tools are for the people, not the individual • Let others contribute and extend them • Have an open ear/IRC channel • Make it easy to create new tools 78 Tips How to build a tooling culture?
79. Build Internal Tools http://www.flickr.com/photos/zzpza/3269784239
80. Enable a tooling culture http://www.flickr.com/photos/usacehq/4920584963
81. Open source needs maintenance Item by LovelyLittleTrove
82. Open source is great Item by SteelPetalPress
83. Thanks! Questions? Harrison Shoff Daniel Schauenberg harry@airbnb.com dschauenberg@etsy.com @hshoff @mrtazz