How to Build a GitHub

Learn about the growth patterns and the architecture behind github.com.

Transcript

1.  githu H O W t B U I L D GITHUB a 
2. githu
3. 6.5MM REPOSITORIES LARGEST GIT HOST 1.9MM USERS SINCE 2008
4. 6.5MM REPOSITORIES LARGEST GIT HOST 1.9MM USERS SINCE 2008 SVN HOST
5. gh  gh  gh  gh  gh  gh 
6. gh  gh  gh  gh  gh  gh  SHOW YOU OUR CARDS going t
7. MAGIC BULLET there i n
8. FOUR STAGES OF GROWTH happiness the EVERYTHING automate
9. NO FORKING HOLMAN @ LOST YO QUIT READING THIS SHIT
10. ho DID WE GIT HERE
11. 1809: PERL INVENTED
12. 1814: COMPUTERS INVENTED
13. 1814-2004: ANARCHY AND CHAOS AND ZOMG EVERYONE’S DYING
14. 2005: VERSION CONTROL INVENTED git
15. 2007: githu GLOBAL PEACE AND HAPPINESS ACHIEVED
16. ...or something like that
17. PRESTON-WERNER TOM GRIT O C TOBER 9, 2 0 07 git via ruby
18. GRIT git via ruby github’s interface to git object-oriented, read/write open source
19. repo = Grit::Repo.new('/tmp/repository') grit repo.commits
20. grit shelling out to git is expensive grit reimplements portions of git in ruby native packfile and git object support 2x-100x speedup on low-level operations
21. grit slowly reimplement grit for speed allows for incremental improvements
22. LED TO GITHUB grit O C TOBER 19, 2 0 07
23. TODAY ADDING 2TB A MONTH 22 FILESERVER PAIRS 23TB OF REPO DATA
24. GITHUB GROWTH THE FOUR STAGES of
25. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:
26. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB: 2008 2009 2010 2012
27. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:
28. JAN 2008 DEC 2008 FOUR STAGES OF GROWTH GITHUB: 42,000 USERS 
29. JAN 2008 DEC 2008 FOUR STAGES OF GROWTH GITHUB: 80,000 REPOSITORIES 
30. LOCAL MULTI-VM SHARED GFS MOUNT
31. LOCAL MULTI-VM WEB FRONTENDS BACKGROUND WORKERS
32. LOCAL MULTI-VM SIMPLE ARCHITECTURE HORIZONTALLY SCALABLE-ish
33. LOCAL SHARED GFS MOUNT SHARED MOUNT ON EACH VM SIMILAR PRODUCTION + DEVELOPMENT ACCESS ALLOWED LOCAL ACCESS VIA GRIT
34. SIMPLE APPROACH, COMMON GIT INTERFACE, QUICK TO BUILD AND SHIP LOCAL
35. LOCAL NETWORKED FOUR STAGES OF GROWTH GITHUB: NET-SHARD GITRPC
36. 2008 2009 2010 FOUR STAGES OF GROWTH GITHUB: 166,000 USERS 
37. 2008 2009 2010 FOUR STAGES OF GROWTH GITHUB: 484,000 REPOSITORIES 
38. the problem: is slow GFS performance degraded as repos added
39. the problem: i/o-bound we’re read/write to disk needs to be fast
40. THE PLAN NETWORKED HARDWARE MOVE DATACENTERS
41. NETWORKED HARDWARE bare metal servers 16 machines 6x RAM machine roles solid datacenter got dat cloud
42. NETWORKED FRONTENDS FILESERVERS AUX DB LAUNCH: SERVER PAIRS
43. NETWORKED GRIT IS LOCAL NEEDS TO BE NETWORKED
44. NETWORKED smoke service is run on each fs; facilitates disk access chimney routes the smoke, stores routing table in redis stub local grit calls, retain API usage, but send over network
45. NETWORKED server pairs offer failover via DRBD real servers, real big RAM allocations
46. NETWORKED LATENCY networked routing adds 2-10ms per request optimize for the roundtrip smoke contains smarter server-side logic
47. NETWORKED LATENCY smoke has custom git extension commands git-distinct-commits returns commits only contained on a given branch calls to git-show-refs and git-rev-list run all calls server-side in one roundtrip
48. NETWORKED HORIZONTALLY-SCALABLE, LATENCY- CONSIDERATE, API-COMPATIBLE WITH GRIT
49. LOCAL FOUR STAGES OF GROWTH GITHUB: NET-SHARD GITRPC NETWORKED
50. 2008 2009 2010 2011 FOUR STAGES OF GROWTH GITHUB: 510,000 USERS 
51. 2008 2009 2010 2011 FOUR STAGES OF GROWTH GITHUB: 1.3MM REPOSITORIES 
52. the problem: duplication data each fork is a full project history 
53. duplication data  i create a repo you fork my repo fs5:/data/repositories/6/nw/6b/de/92/1/1.git fs7:/data/repositories/4/na/3b/dr/72/2/2.git
54. duplication data  1,000 commits 1,001 commits 10MB 10MB 20MB total disk }
55. duplication data  1,000 commits 1 commit 1KB 10MB 10MB total disk }GOAL:
56. duplication data  75 MB repo 3.5k forks x ~250 GB x 2 fs pairs + offsite backups
57. NET-SHARD shard by repository network (“forks”)
58. NET-SHARD network.git 1.git 2.git 3.git 4.git CONTAINS DELTA }CONTAINS ALL REFS ›
59. NET-SHARD network.git GIT ALTERNATES store git object data externally to repository we fetch refs into your fork, transparently
60. NET-SHARD network.git PRIVACY potential leaking of refs cross-network net-shard enabled on all-public and all-private repository networks only
61. NET-SHARD network.git DISK halves disk usage increase disk and kernel cache hits
62. NET-SHARD network.git MIGRATION gradually transitioned repos to network.git effectively feature-flagged by repo
63. NET-SHARD SAVE DISK, IMPROVE PERFORMANCE
64. LOCAL FOUR STAGES OF GROWTH GITHUB: GITRPC NETWORKED NET-SHARD
65. 2008 2009 2010 2011 2012 FOUR STAGES OF GROWTH GITHUB: 1.2MM USERS 
66. 2008 2009 2010 2011 2012 AUGUST FOUR STAGES OF GROWTH GITHUB: 1.9MM USERS 
67. 2008 2009 2010 2011 2012 FOUR STAGES OF GROWTH GITHUB: 3.4MM REPOSITORIES 
68. 2008 2009 2010 2011 2012 AUGUST FOUR STAGES OF GROWTH GITHUB: 6.5MM REPOSITORIES 
69. the problem: GRIT git via ruby
70. the problem: local, ruby-based grit ended up in a high-traffic distributed system
71. the problem: inelegant code spread out everywhere
72. GITRPC network-oriented library for git access GitRPC
73. GITRPC open source fastest git implementation (C) github-sponsored project bindings for all major languages used in our mac, windows clients
74. GITRPC rugged (RUBY) libgit2 (C) gitrpc (RUBY)
75. GITRPC like smoke, gitrpc aims to reduce latency by reducing roundtrips LATENCY
76. GITRPC operations cached on library level CACHING yank out tons of app-level cache logic
77. GITRPC the move to gitrpc started this summer and will take months MIGRATION gradually replace smoke and grit; avoids a risky deploy
78. FAST AND STABLE NETWORKED GIT ACCESS GITRPC
79. LOCAL NETWORKED NET-SHARD GITRPC FOUR STAGES OF GROWTH GITHUB:
80. identify WHAT’S BROKEN
81. sma CHANGES, FAST DEVELOPMENT
82. realCODE BEATS IMAGINARY CODE
83. EVERYTHING automate automate automate automate automate AUTOMATE automate automate automate automate automate automate
84.     m . manage LOL DEVELOPERS SOFTWARE DEVELOPMENT
85.    m . manage DEADLINES MEETINGS PRIORITIES ESTIMATES
86.    m . manage DEADLINES MEETINGS PRIORITIES ESTIMATES
87.  EVERYONE i A MANAGER
88. AUTOMATE AWAY PAIN DEPLOYMENT RECOVERY DEVELOPMENT
89. DEVELOPMENT automate
90. DEVELOPMENT > ./do-work RUN THIS IN EACH PROJECT: ...AND YOU’RE DONE! loljk
91. DEVELOPMENT YOU CAN AUTOMATE THE PAIN OF DEVELOPMENT
92. SETUP DEVELOPMENT the
93. SETUP DEVELOPMENT the ONE-LINER INSTALLS ALL GITHUB DEVELOPMENT DEPENDENCIES
94.  30 min SETUP DEVELOPMENT the CLEAN MACHINE TO FULL DEVELOPMENT ENVIRONMENT
95. SETUP DEVELOPMENT the NEW EMPLOYEES SHIP THEIR FIRST WEEK 
96. SETUP DEVELOPMENT the PUPPET HANDLES ALL DEPENDENCIES
97. DEPLOYMENT automate
98. DEPLOYMENT REAL BROGRAMMERS DEPLOY WITH NO FEAR SO FUCK THAT
99. DEPLOYMENT DEPLOYS SHOULD BE CAUTIOUS, COMMONPLACE, AND AUTOMATED
100. DEPLOYMENT GITHUB DEPLOYS 20-40 TIMES A DAY
101. DEPLOYMENT PUSH BRANCH DEPLOY BRANCH EVERYWHERE · MACHINE CLASS · SPECIFIC SERVERS HUBOT RUNS TESTS IN ABOUT 200 SECONDS USUALLY OPEN A PULL REQUEST
102. DEPLOYMENT DEPLOY LOCKING CAN’T DEPLOY IF A BRANCH IS DEPLOYED AUTODEPLOYS PUSHED TO MASTER WITH GREEN TESTS? DEPLOY.
103. DEPLOYMENT STAFF-ONLY FEATURE FLAGS LIMITS EXPOSURE · REAL-WORLD · AVOIDS MERGES
104. RECOVERY automate
105. RECOVERY SOMETHING WILL ALWAYS BREAK
106. RECOVERY HUBOT IS A SYSADMIN
107. RECOVERY HUBOT LOAD HUBOT QUERIES HUBOT CONNS SERVER LOAD RUNNING DB QUERIES ALL OPEN CONNECTIONS
108. RECOVERY HUBOT RESTORE <REPO> HUBOT PUSH-LOG <REPO> HUBOT GH-EACH <HOST> <COMMAND> RESTORE A REPO FROM BACKUPS SEE RECENT PUSH LOGS TO A REPO RUN COMMAND ON SPECIFIC HOSTS
109. HIGH-LEVEL OVERVIEW IN MINUTES SPEND MORE TIME FIXING AND LESS TIME INVESTIGATING RECOVERY
110. — happiness the — — — —
111. EMPLOYEES HAVE QUIT YEARS 5 EMPLOYEES 108 ZERO
112. 1-2 MONTHS HIRE 1-3 MONTHS RAMP-UP 2 WEEKS LEAVE
113. LOSING AN EMPLOYEE CAN SET YOU BACK HALF A YEAR
114. remove ANY REASON TO LEAVE — — — — — — — — — — — — — — — — —
115. TDD✓ PAIR PROGRAMMING ✓ BDD ✓ TEST-FIRST ✓ DESIGN-FIRST ✓ (just kidding) EMACS x NONE OF THESE ✓
116. WE CARE ABOUT THE WORK YOU DO, NOT ABOUT HOW YOU DO IT
117. LOCATION  HOURS  DIRECTION 
118. LOCATION  HOURS  DIRECTION  GITHUB EMPLOYEES WORK REMOTELY ⅔
119. LOCATION  HOURS  DIRECTION  FAMILY RELOCATION, TRAVEL FREEDOM 
120. LOCATION  HOURS  DIRECTION  CHOOSE YOUR SCHEDULE CHOOSE YOUR VACATIONS FRESH, CREATIVE EMPLOYEES 
121. LOCATION  HOURS  DIRECTION  YOU HACK ON THINGS THAT INTEREST YOU REDUCES BURNOUT 
122. flexible LOCATION  HOURS  DIRECTION  BE TOWARDS WORK/LIFE
123. githu
124. basica y, MOVE FAST = SMALL CHANGES
125. basica y, BE STABLE = DEPLOY CONSTANTLY
126. basica y, HAPPY COMPANY = HAPPY EMPLOYEES
127. thank
128. NO FORKING HOLMAN @ LOST YO QUIT READING THIS SHIT ZACHHOLMAN.COM/TALKS