Services and Rails: The Shit They Don't Tell You (RailsConf 13 Edition)

Building services and integrating them into Rails is hard. We want smaller Rails apps and nicely encapsulated services, but services introduce complexity. If you go overboard in the beginning, you're doing extra work and getting some of it wrong. If you wait too long, you've got a mess.
At Yammer, we constantly clean up the mess that worked well in the early days, but has become troublesome to maintain and scale. We pull things out of the core Rails app, stand them up on their own, and make sure they work well and are fast. With 20+ services, we've learned some lessons along the way. Services that seem clean in the beginning can turn into development environment nightmares. Temporary double-dispatching solutions turn into developer confusion. Monitoring one app turns into monitoring a suite of apps and handling failure between them.
This talk looks at our mistakes and solutions, the tradeoffs, and how we're able to keep moving quickly. Having services and a smaller Rails codebase makes for scalable development teams, happier engineers, and predictable production environments. Getting there is full of hard decisions -- sometimes we get it right, sometimes we get it wrong, but we usually have a story to tell.

Transcript

1. Serviіces and Raiіls The Shiіt They Don’t T ell You
2. Who iіs thiіs guy?
4. •  Over 100 web, 40 memcache servers •  ~500 miіlliіon requests/day to Raiіls app •  98% cache hiіt rate •  ~240,000 memcache gets/sec •  ~20 biіlliіon memcache gets/day Fun Facts
5. •  Test suiіte takes ~10 hours •  JЈenkiіns paralleliіzed across 20 EC2 iіnstances, takes ~20 miіn NOT FUN FACTS
6. But first…
9. •  We have a huge fuckiіng Raiіls app •  300+ models, 200 controllers •  20+ JЈVM serviіces •  Some serviіces do over a biіlliіon requests per day Alriіght, let’s contiіnue
10. …but we still have a huge Rails app.
11. Service Oriented Architectures
12. Components that scale individually.
13. Reusabiіliіty
14. •  Easiіer to scale out serviіce •  Known and prediіctable usage and performance patterns •  Monoliіthiіc archiіtecture means resources for everythiіng Scalabiіliіty
15. •  Smaller and more focused •  Encapsulated concerns •  Push updates iіndependently •  Change out everythiіng wiіthout telliіng anyone Loose coupliіng
16. Codebases that scale organizationally.
17. •  Enabled by loose coupliіng •  Assiіgn a team to the serviіce •  Two teams coordiіnate/agree on APIІs •  Use dummy components Diіstriіbuted executiіon
19. Conway’s Law
20. “Organiіzatiіons whiіch desiіgn systems ... are constraiіned to produce desiіgns whiіch are copiіes of the communiіcatiіon structures of these organiіzatiіons.” Conway’s Law Melviіn Conway
21. The Messagiіng Team
22. •  Deciіdes on iіnterface and iіmplements •  Has siіloed knowledge of system •  Not most iіmportant thiіng to be workiіng on •  Do we keep creatiіng feature teams? The Messagiіng Team
23. Raiіls and Core Serviіces Teams
24. Cross-‐functional teams
25. Example cross-functiіonal team
26. •  Spiіns up, learns domaiіn •  Takes advantage of diіstriіbuted executiіon •  Team coordiіnates on APIІ between serviіces and the cliіent Cross-functiіonal Teams
27. •  Have some cost wiіth not haviіng siіloed “experts” •  Be careful not to couple the APIІ iіmplementatiіon to the cliіent •  After projјect iіs completed, we stiіll have to support the feature Tradeoffs
29. Put them all behiіnd Raiіls
30. Or talk diіrectly to a serviіce
31. There are lots of ways to do iіt
33. ActiіveRecord iіs awesome •  Callbacks and hooks •  Valiіdatiіons •  State machiіnes •  Cache iіnvaliіdatiіon*
34. Ugh, ActiіveRecord •  Ugh, callbacks and hooks •  Ugh, valiіdatiіons •  Ugh, state machiіnes •  Ugh, cache iіnvaliіdatiіon
35. •  Don’t use ActiіveRecord? •  Use your serviіces as iіndexes – jјust store the IІDs •  Move the data so the serviіce owns iіt What are my optiіons?
37. Moviіng data •  Chances are you can’t have downtiіme to move data •  What iіf the serviіce doesn’t work as iіntended? •  Rolliіng back iіs hard, but you need a backup plan
38. •  Backfiіll data to serviіce •  Wriіte data to database and POST to serviіce •  Serviіce can be moniіtored and profiіled •  Move to the serviіce iіncrementally Double diіspatchiіng
39. Now you have duplicate data.
40. The old way is comfortable.
43. •  Close to productiіon •  Runniіng serviіces locally •  Keep up wiіth rapiіdly changiіng serviіces •  Shouldn’t need iіntiіmate knowledge Development enviіronment
45. Deployiіng serviіces •  Need a system that allows you to add new serviіces quiіckly •  Easy to deploy each serviіce •  Maiіntaiіn stable and pre-release packages for development envs
47. Moniіtoriіng and alertiіng •  Performance moniіtoriіng •  Capaciіty planniіng and queue depths •  Reachabiіliіty from dependent cliіents
50. Standardized tools
51. Dropwizard dropwizard.codahale.com
54. Serviіce tradeoffs •  Handle unavaiіlabiіliіty •  Transactiіons aren’t free •  APIІs are much harder to change •  There are no atomiіc deploys
55. Complex systems fail
57. Let’s recap.
58. Always reevaluate your costs and their viability.
59. Organization that supports building services.
60. Tools that allow you to keep moving fast.
61. Be ready to be wrong.
62. Brian Morton / @brianxq3 eng.yammer.com