Paperless Ops Chef Workflow

A short talk given at the January 2013 NYC Chef meetup at Paperless Post HQ. This is a brief overview of how the Paperless Post ops team has formulated it's chef worfklow and the good and bad parts of it.

Transcript

1. Bootstrap the world, homey At first it was beautiful
2. Then the first development change “leaked” to production
3. The setup
4. The setup • 5 Staging environments (1 per team, 1 pre-prod)
5. The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members
6. The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes
7. The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef
8. The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud”
9. The setup • 5 Staging environments (1 per team, 1 pre-prod) • 4 Full time Ops team members • > 10 Developers who should be able to make changes • Many different applications and technologies being managed by chef • Private VMWare/VSphere backed “cloud” • Hosted chef
10. $ /bin/knife kill me --now
11. The AGONY • Managing multiple versions of a cookbook for different developers each environment • Wanting to make and test small changes on different environments • Lack of visibility when deploying changes • Impossible to correlate CHANGES (git) with STATE (chef-server)
12. We want to work like everyone else works
13. $ git checkout -b new_feature # do some work $ git commit -am “Did dat” $ git push -u # pushed to quirkey/new_feature # Pull request/code review $ pp rails deploy earth quirkey/new_feature # Deploys to staging # Sends alert via email + campfire # runs deploy in viewable Jenkins environment
14. But it’s how we work Not just cause it’s pretty
15. Make it work. Start with the API
16. $ pp chef deploy earth johnnydtan/graphite_cluster # git merging # cookbook testing # cookbook uploading # environment munging
18. checkout the staging branch (earth)
19. checkout the staging branch (earth) git merge the deploying branch(es)
20. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging
21. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server
22. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json
23. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env
24. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins
25. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file
26. checkout the staging branch (earth) git merge the deploying branch(es) check for cookbooks changed from the last deploy to staging check latest versions of cookbooks in chef server bump version in metadata and environment.json commit and git push to env send deploy to jenkins jenkins uploads cookbooks and environment file sends notifications to campfire/email
28. The Minor Discomforts
29. The Minor Discomforts • Dealing with conflicts is hard (especially in metadata)
30. The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless
31. The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well
32. The Minor Discomforts • Dealing with conflicts is hard (especially in metadata) • MANY MANY VERSIONS (0.3.192) aka version numbers are meaningless • Doesn’t handle changes to roles (you cant scope a role to an environment) or new cookbooks very well • Chef server consistency is problematic
33. To improve • More code review as part of the process • Simple roles that map to recipes so that roles can be managed/tested at the env level • Using secondary store to keep track of versions (Zookeeper) • Chef versions could be tied to SHA instead of SemVer # (??)
34. github.com/quirkey twitter.com/aq quirkey.com paperlesspost.com THANKS!