Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Puppet in the Trenches: Guerilla DevOps at Brai...

Paul Hinze
September 27, 2012

Puppet in the Trenches: Guerilla DevOps at Braintree

This talk will be a straight-dope report of our real-world experiences using Puppet to manage our infrastructure over the last 3 years. We run Puppet in masterless mode to manage the environments that support our payments processing platform [1].

Our domain provides especially stringent requirements when it comes to security and availability, and I'll be speaking about our practices around Puppet that support these requirements.

When you're in the trenches, it's not always pretty. I'll spend some time talking about the spectacular ways we've screwed up with Puppet over the past few years. I'll link each of those war stories with lessons we've learned about good Puppet practices. I'll provide a few examples of where we feel like applying these lessons have yielded some cool modules (which will hopefully be open sourced and pushed to the Forge by PuppetConf).

There will be no stunning conclusions. As I get up to the more recent lessons we've learned, I'll talk about a few of our unsolved problems, where we run up against the edges of what Puppet itself is meant for as a tool. I'll talk about unanswered questions around where truth should be managed, and outline the ideas we have about where we're headed. In the end, I'll be plugging for the Puppet community as a place for us to collaboratively solve the Hard Problems of managing infrastructure.

[1] https://www.braintreepayments.com/

Paul Hinze

September 27, 2012
Tweet

More Decks by Paul Hinze

Other Decks in Technology

Transcript

  1. master puppet puppet master puppet agent puppet agent puppet agent

    who am i? a proxy server! a db box! an app node!
  2. supply_drop $ mkdir infrastructure && cd infrastructure [infrastructure] $ gem

    install supply_drop Successfully installed capistrano-2.13.4 Successfully installed supply_drop-0.11.1 [infrastructure] $ capify . [done] capified! [infrastructure] $ vim config/deploy.rb let's get
  3. # config/deploy.rb require 'rubygems' require 'supply_drop' def tasks_for_datacenter(datacenter, servers) task(datacenter)

    do role :server, *servers end servers.each do |server| task(server) { role :server, server } end end supply_drop let's use
  4. # site.pp node 'app01.qa' { package { 'python': ensure =>

    installed } } supply_drop let's use
  5. supply_drop let's use [infrastructure] $ cap app01.qa puppet:noop ... notice:

    /Stage[main]/Package[python]/ensure: current_value absent, should be present (noop) [infrastructure] $ cap app01.qa puppet:apply ... notice: /Stage[main]/Package[python]/ensure: ensure changed ‘purged’ to ‘present’
  6. fine grained control $ cap db01.qa puppet:noop # single boxes

    $ cap app{01..05}.qa puppet:noop # classes of boxes $ cap qa puppet:noop # full environment $ git branch --list * master # maps to QA environments * staging # maps to pre-prod environments * production # maps to production environments we always read the diff
  7. parallelization anybody using qa? working on a nagios check i'm

    tweaking the postgres configs, but we should be fine $ cap monitor01.qa puppet:noop $ cap monitor01.qa puppet:apply $ git pull --rebase $ git commit $ cap db01.qa puppet:noop $ cap db01.qa puppet:apply $ git pull --rebase $ git commit
  8. parallelization Squad! Prepping a merge to staging! $ git co

    master && git pull $ git co staging && git pull $ git merge master $ git log origin/staging..HEAD $ git diff origin/staging..HEAD # review git log and diff $ cap staging puppet:noop # review puppet noop output $ cap db01.stg puppet:apply # selectively apply high-risk changes $ cap staging puppet:apply # flush out environment $ cap staging puppet:noop # double check "clean noop" $ git push roger! ten-four!
  9. . ├── config # capistrano tasks & supply_drop config ├──

    datacenters # variable farms inherited by all nodes │ ├── qa.pp │ ├── staging.pp │ └── production.pp ├── manifests # legacy manifests; here be dragons ├── modules # new-style modules ├── nodes # node definitions │ ├── qa │ │ ├── app.pp # divided into per-class files │ │ ├── db.pp │ │ └── proxy.pp │ ├── staging │ └── production ├── puppet.pp # top-level puppet file ├── templates # legacy templates; here too be dragons └── vendor # vendored puppet pushed to machines ├── facter-1.6.4 ├── puppet-2.6.13 └── supply_drop-0.10.2
  10. <VirtualHost *:443> # apache2.conf.erb <% if rails_env == 'prod' -%>

    HostName www.example.com <% else -%> Hostname qa.example.com <% end -%> </VirtualHost> deceptively simple node "app1.qa" { $rails_env = "qa" include users include apache include rails } node "app1.prod" { $rails_env = "prod" include users include apache include rails }
  11. mind your abstractions node "app1.qa" { include apache apache::site {

    'example': hostname => 'qa.example.com' } } node "app1.prod" { include apache apache::site { 'example': hostname => 'www.example.com' } }
  12. mind your abstractions 0..1 or 0..n? ☑ generic or unique?

    ☑ what changes? ☑ required variables ☑ optional variables ☑ defaults ☑