PuppetConf 2016: Scaling Puppet and Puppet Culture at GitHub

@kpaulisse - Scaling Puppet and Puppet Culture ! PUPPETCONF 2016
1 Scaling Puppet and Puppet Culture Kevin Paulisse SRE @ GitHub

About Kevin Paulisse 2 " ! Text @kpaulisse kpaulisse [email protected] x # ↑ Work: GitHub ↓ Live: Madison, Wisc.

Agenda 3 ! Culture • Ownership • Collaboration • Chaos Tools • Workﬂow • Technical debt • Productivity

Puppet at GitHub 4 !

Puppet at GitHub 5 ! • First Commit: September 8, 2008 (GitHub launched on April 10, 2008) • Puppet Versions: 0.24.x - 4.5.x • Lines of Code: 500,000+ • Puppet Resources: 3,000+ per node

Daily deployments of GitHub.com 6 !

Puppet at GitHub 7 ! of GitHub engineers have contributed to github/puppet of all GitHubbers have committed to github/puppet 35% 67% pull requests are merged each day in github/puppet 15+

Puppet at GitHub 8 !

Ownership 9 !

Who owns your Puppet? 10 ! OPS SILO Ops team owns the Puppet code. Others submit tickets.

Who owns your Puppet? 11 ! OPS SILO Ops team owns the Puppet code. Others submit tickets. WILD WEST Everyone owns the Puppet code. The two extremes

Who owns your Puppet? 12 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. OPS SILO Ops team owns the Puppet code. Others submit tickets.

Who owns your Puppet? 13 ! WILD WEST Everyone owns the Puppet code. SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets.

WILD WEST Everyone owns the Puppet code. Who owns your Puppet? 14 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE

Who owns your Puppet? 15 ! SELF SERVE One group owns the Puppet code but others can modify Hiera data. INNER SOURCE One group maintains the project but others can contribute. OPS SILO Ops team owns the Puppet code. Others submit tickets. WE WERE HERE WE ARE HERE WILD WEST Everyone owns the Puppet code.

Workﬂow 16 !

Eﬀective workﬂow checklist 17 ! A Puppet contributor needs to... Check out the Puppet code Bootstrap a Puppet development environment Run unit tests / CI Run Puppet in no-op mode to see changes Deploy Puppet changes to a host Deploy Puppet changes everywhere (???)

GitHub's Puppet development workﬂow 18 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone (Just like nearly every other project GitHub develops)

Clone and bootstrap 19 ! ./script/bootstrap • Ruby gems (bundler) • librarian-puppet • Pre-commit hooks $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Pull request 20 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Test (CI) 21 ! • puppet spec tests with rspec-puppet • puppet-catalogs for ~50 key roles • puppet-lint to enforce style • puppet-real-host-compile to test facter • puppet-utility to verify project setup $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Review 22 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Review 23 ! Types of review: • Team review • Puppet review $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Branch deploy 24 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Ship it! 25 ! $ Branch % Pull Request & Review ' Test (CI) ( Branch Deploy ) Ship * Clone

Collaboration 26 !

Welcome? 27 ! This repository runs the production infrastructure of github.com and many other services. Any changes that break production infrastructure will page the Hubber on call, interrupt their flow, and possibly wake them up. -- CONTRIBUTING.md in github/puppet, October 2015

Welcome!! 28 ! This repository runs the production  infrastructure of github.com and many other  - services. Any changes that break production  - infrastructure will page the Hubber on call,  - interrupt their flow, and possibly wake them up. + services. We encourage (and expect) Puppet  + contributions from everyone within GitHub  + engineering. Changes made here can break  + production infrastructure, so you need to be  + careful. If you need help or a second pair of  + eyes, please feel free to visit us in the  + `#puppet` channel...

Getting contributors on board 29 !

Engaged contributors 30 !

Anti-pattern of engagement 31 ! "Good programmers know what to write. Great ones know what to rewrite (and reuse)." -- Eric S. Raymond, The Cathedral and the Bazaar Brand new programmers will copy the ﬁrst thing that looks close. But also consider...

Which code is the right code? 32 !

Perpetuating technical debt 33 !

Perpetuating technical debt 34 !

Supporting your contributors - education 35 ! Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor

Supporting your contributors - education 36 ! Our Puppet repo... • Where is it? • How to test? • How to deploy? • Where to get help? Learn Puppet... • Book • Tutorial • Formal training • Hack time + mentor

Supporting your contributors - documentation 37 !

Technical debt 38 !

Technical debt 39 ! Technical debt is the enemy of scalability

Watch the technical debt grow 40 !

Watch the technical debt grow 43 ! "If this host is not in physical data center 1, it must be in AWS!"

Modularize for future growth 44 ! Caution: Over-simpliﬁed pseudo-code!

"Good clever" vs. "Bad clever" 45 !

"Bad clever" example 46 ! ipaddress = '172.17.0.1' WITH DOCKER RUNNING m = 25 fqdn_rand(60) A BETTER WAY

Is your solution "good clever"? 47 ! Evaluate your clever solution: • Will you remember how it works? • Will others understand how it works? • What advantages does this have over the  generally accepted path? • If you were on stage at PuppetConf,  would the audience or ?

Paying down technical debt 48 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality

Paying down technical debt 49 ! • Do you set aside time to work on... • Refactoring • Deprecating • Deleting • Enforcing code quality • Do you praise the engineers who do? • Raises and promotions • Performance reviews

Managing chaos 50 !

Monitoring, metrics, and alerting 51 ! Monitors & Metrics

Monitoring, metrics, and alerting 52 ! Monitors: • Catalog fails to generate • Failing resources on host • Host hasn't reported in Metrics: • Catalog compile time • Catalog run time • CI build time

Sometimes things go horribly wrong 53 ! .puppet panic

Sometimes things go horribly wrong 54 !

Productivity 55 !

Development cycle before catalog analysis 56 !

Agent testing 57 ! MASTER AGENT OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS 1. Agent sends  facts to master 2. Master returns  catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs

My first catalog difference analysis 58 ! for host in $hosts ; do git checkout master puppet master --compile $host > /tmp/old.json git checkout my-changed-branch puppet master --compile $host > /tmp/new.json diff /tmp/old.json /tmp/new.json > /tmp/$host.diff done Caution: Over-simplified pseudo-code!

Catalog testing 59 ! MASTER AGENT 1. Agent sends  facts to master 2. Master returns  catalog to agent 3. Agent applies catalog to system code hiera puppetdb ENC logs OS VERSION CPUS MEMORY DATACENTER MAC ADDRESS DISKS FQDN HW MODEL EC2? IP ADDR FACTS

Developing a catalog diﬀerence tool 60 !

Catalog differences with octocatalog-diff 61 ! Catalog compilation: • Check out "from" and "to" branches • Munge hiera config, ENC, etc. • Facts from PuppetDB • Build catalogs using Puppet Comparison analysis: • Resources added, removed, changed • Display human-readable output

Detecting collateral damage 62 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' }

Detecting collateral damage 63 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.6.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } }

Detecting collateral damage 64 ! class mysql::params { $mysql_port = 3306 $mysql_version = '5.5.20-1' } class profile::frontend { package { 'mysql-client': ensure => $mysql::params::mysql_version, } } 5.6.24-7 5.6.24-7

Running octocatalog-diﬀ across a ﬂeet 65 !

Running octocatalog-diﬀ across a ﬂeet 66 !

Productivity improvements with octocatalog-diﬀ 67 ! With octocatalog-diﬀ

Productivity improvements with octocatalog-diﬀ 68 ! With octocatalog-diﬀ

We have open-sourced octocatalog-diﬀ 69 ! https://github.com/github/octocatalog-diﬀ

70 + , ! CULTURAL TECHNICAL • Ownership • Participation • Manage chaos • Workﬂow • Eliminating technical debt • https://github.com/github/octocatalog-diﬀ x Session summary

Scaling Puppet and Puppet Culture 71 ! @kpaulisse kpaulisse x [email protected] # kpaulisse (puppetcommunity.slack.com) Kevin Paulisse (SRE @ GitHub) https://github.com/github/octocatalog-diﬀ https://linkedin.com/in/kpaulisse

PuppetConf 2016: Scaling Puppet and Puppet Cult...

PuppetConf 2016: Scaling Puppet and Puppet Culture at GitHub

More Decks by Kevin Paulisse

Other Decks in Technology

Featured

Transcript