Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Paperless Ops Chef Workflow

Aaron Quint
January 29, 2013
210

Paperless Ops Chef Workflow

A short talk given at the January 2013 NYC Chef meetup at Paperless Post HQ. This is a brief overview of how the Paperless Post ops team has formulated it's chef worfklow and the good and bad parts of it.

Aaron Quint

January 29, 2013
Tweet

Transcript

  1. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members
  2. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes
  3. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef
  4. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud”
  5. The setup • 5 Staging environments (1 per team, 1

    pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud” • Hosted chef
  6. The AGONY • Managing multiple versions of a cookbook for

    different developers each environment • Wanting to make and test small changes on different environments • Lack of visibility when deploying changes • Impossible to correlate CHANGES (git) with STATE (chef-server)
  7. $ git checkout -b new_feature # do some work $

    git commit -am “Did dat” $ git push -u # pushed to quirkey/new_feature # Pull request/code review $ pp rails deploy earth quirkey/new_feature # Deploys to staging # Sends alert via email + campfire # runs deploy in viewable Jenkins environment
  8. $ pp chef deploy earth johnnydtan/graphite_cluster # git merging #

    cookbook testing # cookbook uploading # environment munging
  9. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging
  10. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server
  11. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json
  12. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env
  13. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins
  14. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file
  15. checkout the staging branch (earth) git merge the deploying branch(es)

    check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file sends notifications to campfire/email
  16. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless
  17. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well
  18. The Minor Discomforts • Dealing with conflicts is hard (especially

    in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well • Chef server consistency is problematic
  19. To improve • More code review as part of the

    process • Simple roles that map to recipes so that roles can be managed/tested at the env level • Using secondary store to keep track of versions (Zookeeper) • Chef versions could be tied to SHA instead of SemVer # (??)