Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Pooler Story

The Pooler Story

Pooler is a generic process pooling OTP application. It was originally designed as a connection pool for Riak and was later repurposed to pool Postgresql connections in Opscode's Hosted Chef web service.

This is the story of the design evolution of pooler and how it helped me understand OTP. The primitives for process linking in Erlang make building an exclusive access pool easy. The task of composing those primitives in a way that follows the OTP design principles is less straight forward for the beginner.

In this talk I'll describe the journey from simple gen_server to
complete OTP application and highlight some lessons learned when running the application under load and integrating it into OTP
releases.

Seth Falcon

March 22, 2013
Tweet

More Decks by Seth Falcon

Other Decks in Programming

Transcript

  1. The Pooler Story https://github.com/seth/pooler 1 In the summer of 2010.

    Coming home from work on the bus with a couple Opscode co-workers, discussing a weekend project I was planning for my backyard. Passenger wonders what in the world we’re working on.
  2. When was the last time you used a saw? 4

    Screws or nails? Type of wood? How should the corners go together? How much sand?
  3. http://www.flickr.com/photos/davidstanleytravel/5282834545/ Each simple feature a pile of complexity 5 it’s

    amazing to watch. There’s a special kind of unbreakable thread that connects a simple feature to a load of complexity.
  4. The Pooler Story Seth Falcon Development Lead Opscode @sfalcon 7

    So this is the story of building a SIMPLE connection pool. and how quickly it become not simple.
  5. Supervisor Driven Design 9 Think about the supervision tree as

    a principal aspect Understand new projects by visualizing the supervision tree.
  6. 10 You start, if you haven’t already, by reading these.

    When you are learning, you can’t focus on supervisors first. You need to build an app
  7. I expect to learn something 12 Going to share some

    discoveries (not my inventions) of what I think are good practices Hoping that it isn’t: you can do all of that with gproc and 3 lines of code
  8. 2010 We need an exclusive access connection pool 13 Once

    upon a time, it was September 2010. Experimenting with Riak. Pool Riak pb client connections and act as cheap load balancer
  9. Maintain a pool of members Track in use vs free

    members 14 Simple. Right? And Erlang gives you all the primitives.
  10. Maintain a pool of members Track in use vs free

    members Consumer crashes, recover member Member crash, replace member Multiple pools Load balancing across pools 15 But there are a few more features we’ll need
  11. Start members asynchronously and in parallel Start timeout? Initial pool

    size vs max Cull unused members after timeout When to add members? 16 and yet a few more. Not as simple.
  12. pooler is a gen_server; calls PoolMember:start_link pooler_sup pooler Version 0

    17 Version 0 is here for illustrating the evolution. Simplest possible thing. members are unsupervised.
  13. Know your processes: where they are; where they’re from Hot

    code upgrade Keep process spawning explicit 21 When everybody is supervised, you can easily find a process and know where it is from.
  14. Know your processes: where they are; where they’re from Hot

    code upgrade Keep process spawning explicit The squid will come after you 22 easier to track down process leaks (which could, over time starve vm of ram)
  15. -module(member_sup). -behaviour(supervisor). -export([start_link/1, init/1]). init({Mod, Fun, Args}) -> Worker =

    {Mod, {Mod, Fun, Args}, temporary, brutal_kill, worker, [Mod]}, Specs = [Worker], Restart = {simple_one_for_one, 1, 1}, {ok, {Restart, Specs}}. member_sup supervises pool members pooler_sup pooler member_sup 24 member_sup embeds MFA for member at init, so for pooling different types of members, need another member_sup
  16. -module(pooler_sup). -behaviour(supervisor). init([]) -> Config = application:get_all_env(pooler), Pooler = {pooler,

    ...}, MemberSup = {member_sup, {member_sup, start_link, [Config]}, permanent, 5000, supervisor, [member_sup]}, Specs = [Pooler, MemberSup] {ok, {{one_for_one, 5, 10}, Specs}}. static child spec starts worker_sup pooler_sup pooler member_sup 26 And finally, how the member_sup is wired into the top-level supervisor in pooler
  17. spawn start_link supervisor:start_child supervisor + simple_one_for_one worker No unsupervised processes

    27 Look for instances of spawn and start_link. Add aa simple_one_for_one supervisor and replace the spawn/start_link calls with supervisor:start_child calls.
  18. Rule 1 satisfied. Version 1 But no multiple pools. pooler_sup

    pooler member_sup 28 The member_sup carries the MFA to start a member of a given type Want each pool to have a member_sup.
  19. simple_one_for_one and supervisor:start_link can be used for supervisors too. pooler_sup

    pooler member_sup_1 member_sup_2 pool_sup 30 Probably not news to you. But very useful.
  20. 31 Here’s the message flow for pooler adding a new

    pool and then adding a new member to the new pool.
  21. Bad News... 2012 They need the new stuff next week

    37 We were using poolboy, but saw lockup of pool under load. This was also found at basho and then fixed via QuickCheck. Bug related to queueing when full, different feature/complexity trade-off. pooler just returns an error when full. No queue. With pooler, no hang under load. But..
  22. Start Up Problems 38 pooler doesn’t know about it’s members.

    But needs member’s apps to start before it. And wanted to keep pool config as static.
  23. pooler has no deps. pooler calls emysql:start_link. Who calls application:start(emysql)?

    39 a problem caused by trying to keep things simple and only use static pool config
  24. -module(your_app_sup). -behaviour(supervisor). init([]) -> Pooler = {pooler_sup,...}, Worker = {your_worker,...},

    Restart = {one_for_one, 1, 1}, {ok, {Restart, [Pooler, Worker]}}. in your app: 42 and then start pooler’s top-level supervisor somewhere in your supervision tree.
  25. in pooler: take care with application:get_env 43 application_get_env/1 infers the

    application which will change if used in included_application context. application_get_env/2 is unambiguous so you know where code will look for config. config should be name spaced so /2 is better all around. (20 min mark)
  26. Cast is crazy, so call me (maybe) 46 return_member was

    a cast. Chosen as an optimization. Can end up overwhelming pooler’s mailbox.
  27. When in doubt, call Back pressure avoids overwhelming mailbox 47

    Don’t optimize with cast without measuring. If you know deadlock isn’t a concern, try call first If call isn’t fast enough, consider redesign, not cast
  28. Members started in-line with pooler server loop Slow member start

    triggers timeout 50 Under extreme load and certain error conditions within the system (not pooler in isolation) default timeouts for gen_server:call result in falling off a cliff of failure.
  29. call +∞ Run slower Degrade with load But still run

    51 Changing to call with infinity gives (somewhat) more graceful degradation under failure and avoids some death spiral scenarios.
  30. http://www.flickr.com/photos/yourbartender/5379244544/sizes/l/in/photostream/ 2013 In production at Opscode 53 pooler used in

    production to pool postgres db connections in Opscode Private, Hosted, and Open Source Chef Servers.
  31. Single gen_server serving all pools 56 Doesn’t fit our our

    evolved use cases. Want to pool different things pg and redis. Want isolation.
  32. Can’t dynamically add pools 57 When pooling different things, adding

    pools at run time makes sense. Also solves the startup ordering problem. pooler should be more of a generic service. runs in the background.
  33. In-line synchronous member start 58 want improved dynamic pool growth

    -- add a batch, not just one minimize impact on perf for slow starting members and member crashes
  34. 61

  35. Create supervisors dynamically (take 2) 62 We did this already

    where we used a simple_one_for_one pattern to start new supervisors.
  36. new_pool(Config) -> NewPool = pooler_config:list_to_pool(Config), Spec = pool_sup_spec(NewPool), supervisor:start_child(?MODULE, Spec).

    pool_sup_spec(#pool{name = Name} = Pool) -> SupName = pool_sup_name(Name), {SupName, MFA, ...}. 65
  37. 69 Actual async member start uses starter_sup and a single

    use starter gen_server which triggers member start by setting timeout value to 0 in return from init/1. After creating member and sending msg to appropriate pool, starter exits normally.
  38. async + parallel start (once running) but at init time,

    we want N 71 good for adding capacity dynamically. does not help at pool initialization time
  39. do_start_members_sync(Pool, Count) -> Parent = self(), Pids = [ launch_starter(Parent,

    Pool) || _I <- lists:seq(1, Count) ], gather_pids(StarterPids, []). launch_starter(Parent, Pool) -> Fun = ..., proc_lib:spawn_link(Fun). 72
  40. do_start_members_sync(Pool, Count) -> Parent = self(), Pids = [ launch_starter(Parent,

    Pool) || _I <- lists:seq(1, Count) ], gather_pids(StarterPids, []). launch_starter(Parent, Pool) -> Fun = ..., proc_lib:spawn_link(Fun). Think of the children! 73 Adding async + parallel member start should be easy. This is Erlang after all.
  41. 77

  42. 78

  43. true multi pool all supervised dynamic pool size add batches

    start timeout dynamic pool creation 82 New version now on master. Still a few finishing touches to make some of the dynamic and async features tunable (start timeout, e.g.) Not tagged yet for release, but expected in next couple of weeks.
  44. Take Away • Supervisor Driven Design • No unsupervised processes

    • Create supervisors on the fly • zero timeout in init trick • raw send/receive in init http://www.flickr.com/photos/joeshlabotnik/321872649/sizes/z/in/photostream/ 83