Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Feature Flagging your Infrastructure for Fun an...

Feature Flagging your Infrastructure for Fun and Profit

Presentation from the September 2013 NYC LOPSA meetup about Chef workflow at Etsy and our feature flagging/whitelisting library for rollouts in Chef

Daniel Schauenberg

September 11, 2013
Tweet

More Decks by Daniel Schauenberg

Other Decks in Technology

Transcript

  1. Etsy Infrastructure •~1400 nodes •~30 dev & ops engineers making

    changes regularly •Open Source Chef server + GitHub Enterprise •Default environment setup (production, development, testing) Tuesday, September 10, 13
  2. knife-spork •Work!ow to manage cookbook and environment changes •Versioned cookbooks

    and pinned environments •Speci"c work!ow, di#erent ways of using plugins Tuesday, September 10, 13
  3. % chef-shell chef > recipe_mode chef:recipe > echo off chef:recipe

    > include_recipe "apache" chef:recipe > run_chef Tuesday, September 10, 13
  4. % knife spork check apache % knife spork bump apache

    % git commit % git push % knife spork upload apache Staging Deploy Tuesday, September 10, 13
  5. jonlives/knife-!ip % knife node flip node.etsy.com testing % knife role

    flip testRole testing Tuesday, September 10, 13
  6. % knife spork promote apache % git commit % git

    push % knife spork promote apache --remote Production Deploy Tuesday, September 10, 13
  7. 19:18:06 irccat | CHEF: Daniel Schauenberg promoted [email protected] to development

    https:// github.etsycorp.com/gist/12345 Tuesday, September 10, 13
  8. etsy/chef-handlers 19:20:00 irccat | Chef run failed on test.etsy.com 19:20:00

    irccat | https://github.etsycorp.com/ gist/12347 jgoulah/knife-lastrun % knife node lastrun test.etsy.com Tuesday, September 10, 13
  9. Downsides •Longer testing blocks others •Staged cookbooks can accidentally be

    promoted •Testing environment a#ects more than one cookbook •Used “upgrade” environments to circumvent Tuesday, September 10, 13
  10. Branching in Code •Well established pattern for “dark launches” •Used

    in the Etsy Web stack •Allows for restricted roll outs •http://code.!ickr.net/2009/12/02/!ipping-out/ Tuesday, September 10, 13
  11. chef-whitelist •Data bag driven whitelist •Library to include in cookbooks

    •Easy access to feature !ags Tuesday, September 10, 13
  12. whitelist data bag { "id": "my_whitelist", "patterns": [ "host.example.com", "*.subdomain.example.com",

    "prefix*.example.com" ], "roles": [ "Webserver", "DatabaseServer" ] } Tuesday, September 10, 13
  13. real world example { "id": "php-5-4-19", "patterns": [ "dschauenberg.vm.dev.etsy.com", "web0270.etsy.com",

    "api04.etsy.com", "imgcache01.etsy.com", "imgwriter01.etsy.com", "worker01.etsy.com", "beacon01.etsy.com", "paymentsweb01.etsy.com" ], "roles": [ ] } Tuesday, September 10, 13
  14. Advantages •Easy to access list of what gets upgrades •Upgrades

    don’t need the spork work!ow •Pattern already known by all engineers Tuesday, September 10, 13
  15. Downsides •Changes outside the regular work!ow •No graphs (yet) •Less

    visible cleanup required Tuesday, September 10, 13
  16. Summary •GitHub Enterprise, Dev VMs, chef-shell as development environment •Chef

    Server and knife-spork as Deployment System •Feature !agging with chef-whitelist •Monitoring, Noti"cations, Graphs Tuesday, September 10, 13