Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Automate or Die

Aaron Quint
January 18, 2013

Automate or Die

Explaining Paperless Post's deploy methodology and how we got there. When I Ship, You Ship, We Ship.

Aaron Quint

January 18, 2013
Tweet

More Decks by Aaron Quint

Other Decks in Programming

Transcript

  1. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) Friday, January 18, 13
  2. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) Friday, January 18, 13
  3. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq Friday, January 18, 13
  4. • Food Obsessed (CodeFoodBar) • Open Source Ruby & JS

    (Sammy.js, et al) • BROOKLYNNNNNNNNN (get hype) • twitter: @aq • github: quirkey Friday, January 18, 13
  5. • Most individual user-generated cards sent in a single week:

    ~2 million • Registered user growth over December 2011: 211% (3.1x) • Revenue growth over December 2011: 135% (2.3x) • Number of commits to main app git repo: 44,545 • Number of commits to chef git repo: 4,447 • Peak traffic served: ~300 Mbps • Number of production deploys across applications: 1,631 • Number of staging deploys across applications: 10,600 • Unique animated gifs posted in Campfire: 3,208 Friday, January 18, 13
  6. - A DevOps Oath I, [YOU], Do solemnly swear to

    help developers solve their problems, quickly and without [too many] yak- shaves and make their lives easier, so they can ship code, so that the company can improve, so that we all can be successful. I will do what it takes to avoid repeated tasks, help unblock the blocked, and generally make things work better. Friday, January 18, 13
  7. • Notifications • One click deploys • Continuous integration •

    Distributed/fast deploys Friday, January 18, 13
  8. • Notifications • One click deploys • Continuous integration •

    Distributed/fast deploys • Zero config Dev environment Friday, January 18, 13
  9. • Notifications • One click deploys • Continuous integration •

    Distributed/fast deploys • Zero config Dev environment • Developer Metrics Friday, January 18, 13
  10. • Notifications • One click deploys • Continuous integration •

    Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking Friday, January 18, 13
  11. • Notifications • One click deploys • Continuous integration •

    Distributed/fast deploys • Zero config Dev environment • Developer Metrics • Deploy History/Tracking • Integrated Troubleshooting Friday, January 18, 13
  12. Deploys are too complicated, so I end up not doing

    them ಠ_ಠ Friday, January 18, 13
  13. $ git checkout staging $ git pull root staging $

    git merge my_branch $ git push root staging $ cap staging deploy:long Friday, January 18, 13
  14. WE COULD • Build a web ui to handle deploys

    • Make a simple script that wraps the common commands • Ignore them, haters gonna hate Friday, January 18, 13
  15. I keep losing my changes or forget to commit ಠ_ಠ

    I want notifications when someone deploys ಠ_ಠ Friday, January 18, 13
  16. I keep losing my changes or forget to commit ಠ_ಠ

    I want notifications when someone deploys ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  17. I keep losing my changes or forget to commit ಠ_ಠ

    I want notifications when someone deploys I want to deploy multiple branches at once ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  18. For developers it’s a familiar interface Lots of things can

    be solved with command line tools Friday, January 18, 13
  19. $ pp rails deploy earth quirkey/branch1 mrb/f2 # pp <app>

    deploy <environment> [branches, ...] Friday, January 18, 13
  20. • Does the git flow (checkout, reset, merge, push) •

    Handles conflicts and stashing changes • Runs the deploy • Sends notifications to email list/ campfire rooms Friday, January 18, 13
  21. • Handles booting local services • Has all powerful boot/restart

    commands • Monitors service status Friday, January 18, 13
  22. • Ruby gem (on internal gemserver) • Self-updating (pp devtools

    update) • Built on top of Thor • Made to be extensible (easy to add apps, tasks, recipes) DEvTOOLS Friday, January 18, 13
  23. I need to deploy but I don’t have ssh credentials

    ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  24. I need to deploy but I don’t have ssh credentials

    ಠ_ಠ It takes too long to run the test builds ಠ_ಠ Friday, January 18, 13
  25. I need to deploy but I don’t have ssh credentials

    ಠ_ಠ It takes too long to run the test builds ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  26. I need to deploy but I don’t have ssh credentials

    ಠ_ಠ It takes too long to run the test builds Are we deploying? What step are we at? ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  27. It just needs to be told what to deploy (It

    has an HTTP API) Jenkins can just run devtools Friday, January 18, 13
  28. Text # On your machine $ pp rails deploy earth

    quirkey/branch1 mrb/f2 # Does git merging, sends request to jenkins # On Jenkins, in a `deploy` job $ pp $APP deploy $ENV $BRANCHES --local # Skips the git stuff, just does the actual # deploy and sends the notifications Friday, January 18, 13
  29. Jenkins • Global visibility (changed the game) • No fear

    of losing network connectivity during a deploy • Single point of orchestration • Being able to see the deploys and builds made the team want to make them faster and better Friday, January 18, 13
  30. What branches are deployed to staging? ಠ_ಠ What just got

    deployed to production? ಠ_ಠ Friday, January 18, 13
  31. What branches are deployed to staging? ಠ_ಠ What just got

    deployed to production? ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  32. What branches are deployed to staging? ಠ_ಠ What just got

    deployed to production? Johnny just said he pushed something, but I can’t find it ಠ_ಠ ಠ_ಠ Friday, January 18, 13
  33. Now we just need to collect the data We unified

    the entry points Friday, January 18, 13
  34. > require 'paperless-devtools' > app = PaperlessDevtools.app :rails > app.tools(:git).current_branch

    #=> "master" > app.tools(:jenkins).start_build("master") #=> send build to jenkins Friday, January 18, 13
  35. devtools makes api calls to pushparty which records deploys in

    postgres then uses devtools to send them to jenkins Friday, January 18, 13
  36. devtools makes api calls to pushparty which records deploys in

    postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty Friday, January 18, 13
  37. devtools makes api calls to pushparty which records deploys in

    postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13
  38. devtools makes api calls to pushparty which records deploys in

    postgres then uses devtools to send them to jenkins which uses devtools to send progress and status to pushparty which uses devtools to announce the completed deploy in campfire Friday, January 18, 13
  39. Extremely easy to see what was merged/ deployed, in what

    state and at what time Now we have DATA Friday, January 18, 13
  40. Working with chef [with multiple team members working on multiple

    projects in multiple staging environments] is a pain in the butt. ಠ_ಠ Friday, January 18, 13
  41. $ pp chef deploy earth johnnydtan/graphite_cluster # git merging #

    cookbook testing # cookbook uploading # environment munging Friday, January 18, 13
  42. We have all this data in graphite, chef, sensu, pushparty,

    but when theres an issue it’s hard to coalate the problems ಠ_ಠ Friday, January 18, 13
  43. Awesome things I don’t have time to talk about •

    pp chef create_node Friday, January 18, 13
  44. Awesome things I don’t have time to talk about •

    pp chef create_node • pp vagrant setup Friday, January 18, 13
  45. Awesome things I don’t have time to talk about •

    pp chef create_node • pp vagrant setup • pushparty bashrc’s Friday, January 18, 13
  46. Awesome things I don’t have time to talk about •

    pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration Friday, January 18, 13
  47. Awesome things I don’t have time to talk about •

    pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs Friday, January 18, 13
  48. Awesome things I don’t have time to talk about •

    pp chef create_node • pp vagrant setup • pushparty bashrc’s • graphite integration • deploy build time graphs • more every day Friday, January 18, 13
  49. Build for your team for right now, not for every

    team at any size Don’t expect to build the perfect system Friday, January 18, 13
  50. Not the other way around Start with a workflow then

    tool around it Friday, January 18, 13
  51. Then add the cool shit that they didn’t know they

    needed Build what your team needs first Friday, January 18, 13