Large Scale Continuous Deployment at GitHub

Continuous deployment and its preceding steps of continuous integration, code review, coding, testing, and design are done in a very atypical way at GitHub. These techniques result in developer happiness and a culture of innovation while supporting notable levels of uptime and stable application behavior. In the land of GitHub, chat room-listening robots push binaries to production. Feature toggles dominate the landscape of new user interface and business logic elements. Build metadata injects itself into the version control system. Tickets automatically close themselves based on commit messages and branch promotion. Join Matthew for this presentation and find out the way in which this process emerged, how it is under constant adjustment, what has been working well, where some surprises caused a bit of pain, and what we see as our next steps in this radical but effective way to rapidly ship quality products to customers.

Transcript

1. ! Continuous Deployment 
 at GitHub Large Scale
2. ! matthew@github.com training.github.com @matthewmccull matthewmccullough
7. ! ! setup automation
9. ! ! boxen
10. ! class projects::education-web { github::project { 'education-web': dotenv => true, elasticsearch => true, redis => true, ruby => '1.9.3-p231-tcs-github', postgresql => [ 'education_web_development', 'education_web_test' ] } }
11. Probably
 the most important technical book
 of 2010 -Martin Fowler, ThoughtWorks
12. ! ! Human awareness
13. Hi! I’m Matthew.
14. ! ! flat structure?
15. !
16. GITHUB COMMUNITIES 2770 Most Followed Users - JANUARY 2010 Nodes size : Edges weight : Number of followers Number of forked projects lifo madrobby mislav simonw akitaonrails cv sinatra defunkt rails thoughtbot 37signals drnic ryanb leah wycats mojombo topfunky jnunemaker technoweenie ezmobius aslakhellesoy richhickey rtomayko textmate ericflo binarylogic schacon dhh jamis L l15n l4rk labria lachie lachlanhardy laguiar lak lancecarlson langalex lapis25 laribee larrytheliquid laruby lawrencec lazyatom ldandersen leah leandrosila leedo leehambley leeky leemhenson leeigraham lejoe lenalena lenn0x Leont leshill lestrrat lethain leto leent lfittl lg lhl lholden lian libin lifechurch lifo lincolnloop lindsayeans linkingpaths littleidea liedoor liyanage liz llimllib lllnorikolll lloyd lmorchard localhost lodewijk loe loiane look lopnor lorn loudej lozzd lrug lrz lsdr lsegal lshepard lsmith lucasdecastro lucashungaro luckiestmonkey luddep luigi luislaena luke0x lukebayes lukec lukego lukeman lukemelia lukeredpath lukesh lukhnos luqui lussumo lxbarth lyokato M m0rk m3nt0r m5h maca maccman machine machu Machx macournoyer macsphere maddingue maddox madpilot MadRabbit madridonrails madrobby madx maedana maggit magicalhat MagLe magnolia maiha makenosound makies makoto makotoworld mala malcolmt malkomalko mallipeddi malsup mame mamuso Manfred manuelpichler maneru maoe Marak marcandre marcel marcelinollano marcoow marcpalmer marcusramberg marianoguerra marinho mariusaeriksen markbates markmansour markng markpasc marksands markstory markstos marshall martinbtt martincik martinjandrews martinstannard marucc maryrosecook masaedw masak masaki MASA-P masartz mash masterkain masuidrie mataki mathie matschaffer matsufuji matsuu mattallen mattb mattball mattbauer mattetti mattgallagher matthew matthewford matthewmccullough matthewpennell matthewrudy matthooks mattknox mattly mattman mattmatt mattn mattpolito mattt mattwynne mattyoho matz mauricio maxim maxime maxlapshin maxy mayuki mbailey mbbx6spp mbleigh mblsha mbrubeck mcantelon mchung mcommons mcornick mcroydon mcurry mdesign mdg mdirolf mdk Me1000 melo memcached memo memowe mental merb merbjedi mergulhao meritt merlinmann metade metajack metaskills methodmissing mexpolk mferrier mfp mhinze miamiphp michael michaeld michaelklishin mickey24 microformats mig migueldeicaza migurski mihasya mikeal mikechambers mikehale mikejs mikesten miketracy mikewest miki mikl Miksago milesj mileszs millenomi miloops mintchaos miraglia mischa misla mitchellh mitsuhiko miura1729 miyagawa mizzy mjankowski mjfreshyfresh mjgiarlo mkhl mkoukoullis mkrisher mlaaker mlandauer mlemos mletterle mloughran mmalone mmangino mmcgrana mmower mnaberez mndoci mnot mnutt mocra mojodna mojombo mokehehe mokele mokolabs molpe monadic monde mongodb monjudoh monken monkrb monstersofruby Montoya montylounge moomerman mootoh mootools moritz moriyoshi moro mort motemen mperham mpeters mptre mrdon mreid mrflip mrkn mroth msanders mschilli mschrag mst msweeney m-takagi mtodd muness mutoh m midaurre mxcl myabc myelin myfinder myles mynyml mysociety mza mzarra mzp N n0ts n1k0 N3uro5i5 n8han NachoMan Nafai77 nagayama naha nakajima namelessjon nando nanki nanto naoya naoyat napcs Narnach nataliepo natbat nate nateabele natekoechley natfriedman nathanborror nathanpitman nathansmith nathansobo naeen nawoto nay nayutaya nbio nciagra ncr nddrylliog nealford neerajdotname nefarioustim negipo neilcrookes NeilCrosby neilk nekokak nekoya nelstrom neror nesquena netshade nette neyn newbamboo newism newrelic newtonapple nex3 nfjinjing ngenworks ngerakines nickdunn nickel nickoneill nicksieger nigelr nihen nikodemus nilshoerrmann ninajansen ninjudd nipotan nitobi nitoyon niw nkallen nkohari nkpart nmerouze noblejasper nofxx nogeek nono noplans norbert norio norio-nomura norm norman northern-bites nosmo noss notahat notbenh notch8 nothingmuch no noas0x2a nowells nperez nrk nshah ntalbott ntoll nuance nullobject nullstyle nyarla nyaxt nysenatecio nzakas NZKoz O obie objo obra oc ofZach ogijun okkez olabini oldmoe oleganza olg olleolleolle ollym omega omnigroup omo omz ondrejmirtes onecrayon oneup ook openframeworks openid openjsan openmelody openpne openskip openzoom opscode oquno orderedlist orionz orlando os0x osamu2001 osfameron Oshuma oskarkrawczyk osteele otsune outoftime Oid oz ozeias P pablete packagethief pacoguzman padraic pahagon palewire palm paltman pantulis Paperision3D parolkar particle patmaddox patrickt paukul paulbuchheit paulca pauldix pauldowman paulelliott paulhammond paulirish paulmars paulp paulsmith pc pcalcado pdcawley peas pedro pedromg pelargir peleteiro pelle pellegrino penberg penguincoder pengwynn penso perigrin peritus perl6 petdance pete peterc peterlih peterpunk pfenwick pfig pgr0ss ph7 phae phatduckk phaylon phiggins42 philhawksworth phillipadsmith philnash philogb philoye philsturgeon phishy phonegap phorsfall php phpnut phuang piece pieter pilu pinax piranha Pistos pius piotal pjb3 pjbarry pjhyett pk pkrumins pkwde planetargon planetbeing plataformatec plathrop playpauseandstop pledbrook plentz plu pluginaweek pmichaud pnasrat pneff pobox pointlessjon pointybeard pokeb pope poppen porras postmodern PotHix potionfactory ppeszko practicalarduino pragdae pragmaticbadger predominant presently preston probablycorey processone progit progprog progrium projectblacklight protocool psd psychs ptarjan puls pumpkin puppet-modules pure purzelrakete pydanny Q qhoxie qoobaa qrush quackingduck quag quamen quek queso quirkey R r38y rabbitmq rabble rack rackspace radar radiant raelity rafaelbandeira3 rafaelp rafaelrosafu rafaelss rafaelsteil rafl raganwald raggi rails railsbridge railsdog railsloe railsmachine railsmaniac railsrumble rakudo ralph ramalho Ramarren ramon ramsey RandalSchwartz RandomEtc rapidsms rapodaca rares raul rayners rayinly rbarazi rblackwe rbuels rbxbx rcaputo rcarneiro rcrowley rcunning rdp reagent recess reddais redinger redsquirrel reductielabs reid reidab reidmix reinh releance remi remko A a2chub aalmiray aanand aarongustafson aaronjensen aaronsw abcde abedra aberant abh abi Abigail abock abraham abw acastro acdha acf acme aconbere acotie actionrails Actionscript3Commons actiefx actiescaffold actsasflinn AD7six adambair adamcooke adamhjk adamkennedy adamlogic adammaddox adammck adamstac adam adamwiggins adelcambre adinardi adrianh adzap aeden aemkei AF83 agencyrainford agentzh agile ahoward ahwayakchih ai aitor aiwilliams ajturner akio0911 akisute akitaonrails akiyan akosma akr4 akrito al3x alandipert AlanQuatermain alcides alecmce alex alexbain alexcharlie alexm alexott alexollmer alexyoung alfredwestereld alkemist allen alloy alowe alpinegizmo alto alarobp amachang amaia amartinsn amatsuda amazingsyco ambethia amiel amiri amitrathore amnesiac amyhoy anandkunal anandology anarchiist anatoo andk andre andreisau andreiz andrew andrewgodwin andrewmaier andrewsmedina android AndyA andyferra andyhd andykent andymatuschak anildash anotherjesse anselm anselmoales anteaya anthonyshort antirez antoniogarrote antono anutron ap apache apeiron apeiros apenwarr APE-Project apgwoz apinstein apm apocalypse apotonick apparentlymart appcelerator appden applicake aptana aq1018 aql Arachnid aral arcanez archaelus archfear arfon armmer arnaudsj arneb ars arsenio arthurgeek arthurk arunthampi arind arya asenchi ashb ashbb ashchan ashchristopher ashizawa ashleymoran ashrewdmint ask aslakhellesoy assaf astro atduskgreg atebits atinypixel atl atmos atnan audreyt augustl aurelian auser aussiegeek austinonrails autarch authorNari automatthew auxesis aar adi asm ayende ayn azambuja B bacek backtik baldwindaid balibali balinterdi ballantyne bangpound banister bantic baobab barbie bard bardusco bartman bartTC barttenbrinke bastih bastos batiste bauhouse baardage bayashi bborn bcardarella bclinkinbeard be9 BeamerCola bear beausmith beberlei beebe4 bekkopen benadida benaskins benatkin benburkert bendiken BenHall benhoskings benjaminhawkeslewis benoitc benschwarz benspaulding benstucki BenWard beppu besquared bess bestpractical bfirsh bgolub bguthrie bhorst bickfordb bigfleet bigtiger billy3321 binary42 binarydud binarylogic bingos bitbckt bitprophet bi bjartek bjeanes bjhess bkerley blackwinter blaine bleargh45 blech bleything blowmage blturner Bluebie bluesmoon BlueT BM5k bmabey bmentges bmizerany bobmcwhirter bob-p bobtfish bobuk bomberstudios bonar bonnu book boosty bos botanicus boucher bowserj bradchoate bradfitz bradford bradleywright brainopia braintree brandon brandonaaron brandonkelly breakpointer brendanlim brendano brentsnook bret briandfoy brianjlandau brianleroux brianmario BrianTheCoder bricas bricooke brightbox britg brixen brl BRMatt brosner browsermedia browserplus bru bruce bruno brunofms bruno brupm bry4n bryanl bryansray brycethornton brynary brysonian bs bsag bschmalhofer bscofield bslatkin btakita btrott bumi bwhitman byrnereese C c7 c99koder c9s cadorn caelum caffo Caged caike cainley caironoleto caius cakebaker cakephp cakephp-plugin-proider calagator calaera calendaraboutnothing camflan camptocamp capistrano capndesign capotej capttaco cardmagic careo caridy carlhuda carljm carllerche carlo carlosantoniodasila carlosbrando carmelyne carpeaqua cassiomarques castleproject casualjim caueguerra cauld caalle cayblood cbilson ccgus cchandler cdent cee-dub ceeram cellfusion centro cerego ceritium cesare cgrand cgriego cgrusden chad chadcrowell chadmyers chaoslawful charley charsbar cheeaun chendo chergert chillicoder chipdude chirags chl chneukirchen cho45 choan choonkeat chorny Chouser chregu chris-d-adams chrisdolan chrisdrackett chriseidhof chriseppstein chrisk chrislloyd chrislo chrismear chrismessina ChrisMissal chrispatterson christos chromakode chromatic chromium chubas chunzi chuyeow ciaran ciaranm cicloid citrusbyte cjheath cjmartin cjohansen ckelly claco claesjac clarkware claus clayton cldwalker cleitonfco clemens cliffmoon clintecker clkao clockfort clofresh cloudera cloudhead cloudkick clouserw cmer cmilfont coatl CobyR cocoiti codahale codebutler CodeOfficer codepo8 coderrr codeslinger codyfauser codysoyland cog coji coleifer colin collectieidea collin commanderklaus commondream commonsguy Constellation cooldaemon copiousfreetime corey coreyhaines coreyti cortesi cosimo couchio couchrest courtenay cout cowboy cowens cpennington cpojer cpytel cqa02303 crafterm Craftworks craigw creationix crnixon crossblaim cschneid cstar ct ctb curtis cv cvf cwensel cwninja cwsaylor cyanogen cypher cyrildoussin cyx czheng D da3mon dacort daeken Daenyth dag dagolden daijiro daiki41ti dajobe dakrone dalmaer dambalah damog damon dan DanaDanger dancroak dand danhodos danielharan danielmorrison danielpunkass danielqo danielsdeleo daniellopes danielwanja dankogai danlarkin danlucraft dan-manges dann dannyamey danrubin danski danw danwrong darashi darius DarkAngelBGE darkwing darrenf darronschall darscan darwin datamapper datawrangling daebryson daejohnson daeman692 daemerwin daeott daetroy daglass daidbgk daidjrice daidpersson daidrichards daidsmalley daiscabral daisp daorg dbalatero dbloete dbr dbrady dbussink dcbriccetti dce dcgrigsby dchelimsky dcparker dcramer dcrec1 dcrosta dctanner ddemaree ddfreyne ddollar deadprogrammer dealforest deanwilson debasishg deepak deepthawtz defunkt Def dekart delano delynn denderello derailed derencius derickbailey desi destroytoday deelop deelopmentseed deelopwithpassion dfischer dgiunta dha dhellmann dhh dhoss diegok dies-el Dieterbe digdog digg digi604 digitarald dima dira directeur dioxx dj2 djacobs django django-extensions djangrrl djanowski djchall djspiewak djsun dkubb dmathieu dmitriid dmitryame DmitryBaranoskiy dneg dneighbors DocSaage documentcloud dogmatic dojo dojosp dominiek domix domm donpdonp dorward dot dottedmag dougal dougma doy dpickett dpp draegtun dragon3 drbrain drd drewolson drforr drikin DRMacIer drnic drudge dsboulder dsingleton dstrelau dto dtsato duairc dudemeister duelinmarkers dukedae duncanbeeers durran dustin duwanis ddplm dydra dwhittle dwillis DylanFM dyoder dysinger dziegler earl eastmedia E easy-designs ebi ebihara edas edais10 eddanger U.S. Users Japaneses Users U.K. Users German Users Other Users
17. ! ! –Sheryl Sandburg, COO of Facebook "I'm not sure there's room for
    just management anymore."
18. ! ! Current state of affairs
23. we have a problem
25. "This is so important, we can't entrust it to automation. We need our top developer to supervise it."
27. ! ! King of developers
30. ! ! The loose parts
31. automation everywhere
33. continuous integration
37. ! ! hubot
38. agile practices
39. traceability
40. rollback
41. toggles
44. humane interface
46. ! ! Pull request?
60. ! ! Divide and conquer
65. ! ! Automate everything
68. continuous integration servers
69. continuous integration servers jenkins travis ci circleci buildhive
75. ! ! Embrace transience
76. Amazon Web Services
78. Heroku
80. Hubot
82. hubot: <dosomething>
83. ! ! Measure everything
94. ! ! Feature Flag beta code
95. http://www.flickr.com/photos/mwichary/3209181446/
98. ! ! Fail forward
100. git revert
102. small changes
103. try not to roll back
104. fail forward
105. ! ! Look at the leaders
109. ! ! Embrace risk
110. kill some features
111. discover we're wrong
112. cause some regressions
113. optimize for innovation
114. ! ! –W. Edwards Deming "It is not necessary to change.
     Survival is not mandatory."
115. ! Q&A
116. ! matthew@github.com github.com/training @matthewmccull matthewmccullough