Cross node orchestration with Chef and Noah

long-form of my talk at #chefconf

Transcript

1. Cross-node Orchestration for Chef with Noah
2. Cross-node Orchestration for Chef with Noah (sort of)
3. Let's step back a little
4. We've got this part down { "name": "es_dispatcher_lb", "description": "", "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { "postfix": { "myhostname": "dispatcher-lb.enstratus.com" } }, "chef_type": "role", "run_list": [ "role[es_loadbalancer]", "recipe[enstratus::dispatcher_lb]", "role[monitored_host]" ], "env_run_lists": { } } {
5. We've got this part down { "name": "es_dispatcher_lb", "description": "", "json_class": "Chef::Role", "default_attributes": { }, "override_attributes": { "postfix": { "myhostname": "dispatcher-lb.enstratus.com" } }, "chef_type": "role", "run_list": [ "role[es_loadbalancer]", "recipe[enstratus::dispatcher_lb]", "role[monitored_host]" ], "env_run_lists": { } } Do this Then this Then this
6. Congrats! You've got a load balancer! (now what?)
7. Remember this line? "recipe[enstratus::dispatcher_lb]"
8. The deep Chefster backends = search(:node, "role:#{node[:enstratus] [:dispatcher_lb][:dispatcher_server_role]} AND chef_environment:#{node.chef_environment}") + <% @backends.each do |be| -%> server <%= be.name %> <%= be.ipaddress %>:2023 check inter 5000 <% end -%> = "recipe[enstratus::dispatcher_lb]"
9. The deep Chefster backends = search(:node, "role:#{node[:enstratus] [:dispatcher_lb][:dispatcher_server_role]} AND chef_environment:#{node.chef_environment}") + <% @backends.each do |be| -%> server <%= be.name %> <%= be.ipaddress %>:2023 check inter 5000 <% end -%> = "recipe[enstratus::dispatcher_lb]"
10. More appropriately = haproxy backend backend backend backend
11. Congrats! You've got a distributed system!
12. What happens when I... • Add a new backend? • Remove a dead backend?
13. You want to... • Have confguration changes on one system affect change on another system • Gate confguration changes on one system based on state of another system • Establish relationships between systems
14. But which changes?
15. But... • Should adding a new backend node cause a chef- client run on the haproxy nodes? • If the chef-client run on the haproxy node changes a different setting, what node(s) needs to know that? • What if THOSE nodes change something? NOW who do we tell?
16. Congrats! You've got a graph!
17. It's a graph....
18. It's a graph....
19. It's a graph....
20. It's a graph!
21. Be careful what you wish for • Circular dependencies • Transitive dependencies • Version conflicts • Computational cost
22. So about Noah
23. What is it? • Sinatra + Redis app • HTTP + JSON • RESTish • Watches (callbacks) • Inspired by, but no replacement for, Apache Zookeeper
24. Data goes in... curl -X PUT -d '{"name":"fooapp"}' \ http://noahserverhost:5678/applications/fooapp {"result":"success", "id":"0d52d942-4580-f30e-45154edda2a2fc3d", "action":"create", "name":"fooapp" }
25. Data comes out... curl -X GET \ http://noahserverhost:5678/applications/fooapp { "id":"0d52d942-4580-f30e-4515-4edda2a2fc3d", "tags":[], "links":[], "name":"fooapp", "created_at":"Thu May 10 02:22:53 UTC 2012", "updated_at":"Thu May 10 02:22:53 UTC 2012", "configurations":{} }
26. Primitives/Data Types • Host • Service • Application • Configuration These are “get you started” primitives
27. Ephemerals • Arbitrary paths in the URL scheme namespaced under /ephemerals • Key is path name /ephemerals/foo/bar/chefconf/ • Value is...whatever Doesn't have to be JSON. Could be a blob. • No relation in between keys /foo/bar is unrelated to /foo
28. Watches • “Pluggable” callbacks • Think triggers • Register against any path in the system • Changes at that path (or below it) execute the registered callback • Sends JSON dump of the state of change
29. Watch Example (part 1) • Add some data curl -XPUT -d '1' http://localhost:5678/ephemerals/chef/notify/localhost • Get a response {"action":"create", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"1", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 03:56:25 UTC 2012"}
30. Watch Example (part 2) • Register a watch curl -XPUT -d '{"pattern":"//noah/ephemerals/chef/notify/localhost", "endpoint":"http://localhost:8080"}' http://localhost:5678/watches/ • Get a response {"action":"create","result":"success","id":"51828298-ad79-26c3-459b- fd757b85db75","pattern":"//noah/ephemerals/chef/notify/localhost","name" :"Ly9ub2FoL2VwaGVtZXJhbHMvY2hlZi9ub3RpZnkvbG9jYWxob3N0fGh0dHA6Ly9sb2NhbG hvc3Q6ODA4MQ==","endpoint":"http://localhost:8080,"created_at":"Thu May 10 03:02:25 UTC 2012","updated_at":"Thu May 10 03:02:25 UTC 2012"}
31. What just happened? • We told Noah that when a change happens to /ephemerals/chef/notify/localhost • send an HTTP POST to http://localhost:8080/
32. Now what? • Change that data curl -XPUT -d '2' http://localhost:5678/ephemerals/chef/notify/localhost Get a response {"action":"update", "result":"success", "id":"7e743edd-a40f-5062-a5a0-70a8b4df7721", "path":"/chef/notify/localhost", "data":"2", "created_at":"Wed May 09 03:56:25 UTC 2012", "updated_at":"Wed May 09 04:00:00 UTC 2012"}
33. But something else happened this time... I, [2012-05-09T00:01:00.395491 #36180] INFO -- Noah::Agent: Found new watches D, [2012-05-09T00:01:00.395611 #36180] DEBUG -- Noah::Agent: Current watch count: 0 D, [2012-05-09T00:01:00.396681 #36180] DEBUG -- Noah::Agent: New watch count: 1 I, [2012-05-09T00:01:00.396850 #36180] INFO -- Noah::Agents::HttpAgent: Noah::Agents::HttpAgent worker initiated I, [2012-05-09T23:10:36.438180 #38052] INFO -- Noah::Agents::HttpAgent: Message posted to http://localhost:8080 successfully
34. And what did localhost:8080 see? { "id"=>"7e743edd-a40f-5062-a5a0-70a8b4df7721", "tags"=>[], "links"=>[], "path"=>"/ephemerals//chef/notify/localhost", "data"=>"1", "created_at"=>"Wed May 09 03:56:25 UTC 2012", "updated_at"=>"Thu May 10 03:10:36 UTC 2012", "action"=>"update", "pubcategory"=>"//noah/ephemerals/chef/notify/localhost" }
35. What can we do with that? • Parse the data to effect some change? • Use it as a signaling mechanism? • Alert on it? • More importantly, how can we use it with Chef?
36. Noah cookbook • http://github.com/lusis-cookbooks/noah • Installs Noah + Redis • Adds a noah_get method • Automatically registers node in Noah (noah::register recipe) • Also some new resources....
37. Noah LWRP • Register any primitive • noah_application • noah_service • noah_configuration • noah_application • Dump some data in an ephemeral node • noah_ephemeral • Block the chef run until some data shows up in Noah • noah_block
38. Common Scenario haproxy django1 django2 mysql
39. Current Solution • knife bootstrap -r “role[mysql]” • Wait • knife bootstrap -r “role[django1] • Wait • knife bootstrap -r “role[django2]” • Wait • knife bootstrap -r “role[haproxy]”
40. noah_block haproxy django1 django2 mysql
41. noah_block haproxy django1 django2 mysql
42. noah_block haproxy django1 django2 mysql • Needs a DB • Loads DB Needs to wait for initial DB load Needs backends Needs a DB
43. Django1/2 recipe noah_block "wait_for_db" do path "http://noah:5678/ephemerals/#{db_master_role}" timeout 600 data "ready" retry_interval 5 on_failure :retry end
44. Django2 recipe noah_block "wait_for_migration" do path "http://noah:5678/ephemerals/migration-status" timeout 600 data "done" retry_interval 5 on_failure :retry end
45. haproxy recipe noah_block "wait_for_backends" do path "http://noah:5678/ephemerals/backends" timeout 600 data "available" retry_interval 5 on_failure :retry end
46. Triggering Chef Runs • Use noah_ephemeral at end of the dependency run_list (on the backend node) • Leverage string interpolation for path name • Data is unimportant, just the signal
47. Small Problem... There's no “pokeable” Chef client
49. Small Solution • Small web app • listens for HTTP POST requests • runs chef-client
52. Almost.....
54. Demo time
55. Well that was fun • Obviously needs to be done “right” • Add security • Write your own! • Watches are pluggable. • AMQP • ZeroMQ • HTTPS + Token auth?
56. Wrap up • Questions? • Comments? • Rude remarks?
57. Thanks! github.com/lusis twitter.com/lusis blog.lusis.org github.com/lusis/noah