My Mom told me that Git doesn't scale

Transcript

2. Thanks for being here
4. github
5. github Git hosting: No longer a pain in the ass
6. github Git hosting: No longer a pain in the ass for you. Not for us. Because, goddamnit, if I ever find the guy who invented this thing I’m going to hang him from
7. Let’s host some Git repos!  file.c  src  file.h  README.md  COPYING.md  .git  Bare Repository  HEAD  index  objects  refs git-daemon
8. OK, now about the web... grit Ruby - Git interface
9. OK, now about the web... grit  Bare Repo  Bare Repo  Bare Repo Ruby - Git interface
10. 1VM grit storage rails app
11. nVM storage
12. nVM storage (GFS)
13. Rails was making us slow.
15. Literally.
16. Time to move to Real Hardware
17. fileservers frontends db
18. fileservers frontends db ?????????
19. smoke
21. bert (binary Erlang term)
22. bert (binary Erlang term) ernie (not an acronym)
23. chimney (Redis) frontend fileserver smoke grit ernie grit
24. Horizontal Scaling Vertical Scaling
25. Horizontal Scaling Vertical Scaling problums wit them gigabits problums wit them gigahurtz
26. them gigabits
27. bummer x4180 = A LOT.
28. NetShard       ... alternate network
29. NetShard       ...  alternate network
30. them gigahurtz
31. bottleneck: grit
32. bottleneck: grit solution: git shell out to
33. bottleneck: git shell out to
34. bottleneck: git solution: git shell out to shell out to
35. bottleneck: git solution: git shell out to shell out to properly
36. posix_spawn
37. posix_spawn Seriously. < 1ms
38. posix_spawn Seriously. < 1ms The issue is not in “shelling out”, the issue is in the spawned process.
40. GUISE
41. GUISE GUISE
42. GUISE GUISE GUISE
43. GUISE GUISE GUISE ...what?
45. Why don’t we take
46. Why don’t we take the Git binary...
47. Why don’t we take the Git binary... yeah?
48. Why don’t we take the Git binary... yeah? and compile it as
49. Why don’t we take the Git binary... yeah? and compile it as a library
50. Why don’t we take the Git binary... yeah? and compile it as a library oh... go on...
51. Why don’t we take the Git binary... yeah? and compile it as a library oh... go on... and link that into
52. Why don’t we take the Git binary... yeah? and compile it as a library oh... go on... and link that into our server
53. Scientific Graph™
54. Memory Usage Time Scientific Graph™
55. Memory Usage Time Scientific Graph™
56. Memory Usage Time Scientific Graph™
57. Memory Usage Time Scientific Graph™
58. Memory Usage Time Scientific Graph™
59. Memory Usage Time Scientific Graph™
61. Well, we didn’t think about
62. Well, we didn’t think about freeing memory, but...
63. Well, we didn’t think about freeing memory, but... THIS IS THE KIND OF PROBLEM WE COULD SOLVE WITH CGI
64. Well, we didn’t think about freeing memory, but... THIS IS THE KIND OF PROBLEM WE COULD SOLVE WITH CGI IN 1995
65. Scientific Graph™
66. Memory Usage Time Scientific Graph™
67. Memory Usage Time Scientific Graph™
68. Memory Usage Time Scientific Graph™
69. Memory Usage Time Scientific Graph™
70. Memory Usage Time Scientific Graph™
71. Memory Usage Time Scientific Graph™
72. Memory Usage Time Scientific Graph™
73. What do you mean the server died?
74. die("BUG: non-INDEX attr direction in a bare repo"); What do you mean the server died?
75. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad revision is needed"); What do you mean the server died?
76. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad revision is needed"); die("'%s' is not a valid branch name.", name); What do you mean the server died?
77. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad revision is needed"); die("'%s' is not a valid branch name.", name); die("Empty patch. Aborted."); What do you mean the server died?
78. die("BUG: non-INDEX attr direction in a bare repo"); die("a bad revision is needed"); die("'%s' is not a valid branch name.", name); die("Empty patch. Aborted."); die("unable to read index file"); What do you mean the server died?
79. libgit
80. libgit2 the “2” means this one frees memory
81. libgit2 the “2” means this one frees memory NOT ENOUGH ABSTRACT FACTORIES
82. JGit the “J” means this one is in Java ...not our thing.
83. Java a brief timeline New companies don’t use Java because it’s not like Unix 1995 New companies use Java because it’s new and shiny 1997 New companies don’t use Java because it’s ooooooold 2005 New companies use the JVM because 2011
84. Java a brief timeline New companies don’t use Java because it’s not like Unix 1995 New companies use Java because it’s new and shiny 1997 New companies don’t use Java because it’s ooooooold 2005 New companies use the JVM because 2011 github
85. If you think you understand the JVM, you are either:
86. If you think you understand the JVM, you are either: a) Very smart
87. If you think you understand the JVM, you are either: a) Very smart b) Very wrong
88. If you think you understand the JVM, you are either: a) Very smart b) Very wrong
89. Some people think that github is a Rails shop Ruby shop. or even a
90. Some people think that github is a Rails shop Ruby shop. or even a github is a Unix shop and everything else is just a detail.
91. libgit2 So,
92. libgit2 a brief timeline Shawn Pearce The Past
93. libgit2 a brief timeline Shawn Pearce myself The Past
94. libgit2 a brief timeline Shawn Pearce myself myself (about to have a mental breakdown) The Past
95. libgit2 a brief timeline Shawn Pearce myself myself (about to have a mental breakdown) myself (having a mental breakdown) The Past
96. libgit2 a brief timeline Shawn Pearce myself myself (about to have a mental breakdown) myself (having a mental breakdown) myself (reaching Git nirvana) The Past
97. libgit2 a brief timeline Shawn Pearce myself myself (about to have a mental breakdown) myself (having a mental breakdown) myself (reaching Git nirvana) The Past Russell Belfer Carlos Martín Michael Schubert Ben Straub real contributors
98. libgit2 a brief timeline ?
99. libgit2 a brief timeline ?
100. libgit2 a brief timeline ? 1.0 release
101. libgit2
102. libgit2
103. Good Heavens, just look at the time. It’s NoSQL o’clock NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL NoSQL
105. ...do you even
106. ...do you even mongo?
107. Key-Value Stores The Magic of If you wish upon a star, and have a pure heart...
108. Key-Value Stores The Magic of If you wish upon a star, and have a pure heart... Anything can be a Key-Value store!
109. id name state lat 13 San Francisco CA 24 24 Phoenix AZ 33 7 Denver CO 40 8 Caribou ME 47 2 Los Angeles CA 22 SELECT * FROM CITIES WHERE name = ‘San Francisco’ Key-Value Stores The Magic of
110. Git is queried like a Key-Value Store But it is not a Key-Value store git show f3c896c1949476e85abc0d75bb2143656a9580a6
111. a b r i e f i n t r o d u c t i o n t o t h e G i t d a t a m o d e l
114.  file.c  src  file.h  README.md  COPYING.md
115.  file.c  src  file.h  README.md  COPYING.md tree src/ README.md COPYING.md tree file.c file.h blob blob blob blob
116. commit parent tree T metadata
117. commit T commit T commit T commit T commit T commit T Behold, a graph.
119. Well that was easy.
121. master Oh god kill me
122. Li le known torture methods:
124. warning: the rabbit hole is pretty deep
126. warning: git totally wasn’t designed for this
127. Git doesn’t give a #!%$ about CAP
129. Number of hops on a complex query 1,000,000
130. Number of hops on a complex query 1,000,000 Required hops for a successful query 1,000,000
131. Number of hops on a complex query 1,000,000 Required hops for a successful query 1,000,000 Replica count to ensure 100% availability a metric shitton
132. We could fix it.
133. We could fix it. But we won’t.
134. GitRPC
135. GitRPC Less.
136. GitRPC Rugged libgit2 server Ruby Ruby C
137. chimney (Redis) frontend fileserver smoke grit ernie grit GitRPC GitRPC
138. chimney (Redis) frontend fileserver GitRPC GitRPC server client
139. New serialization protocol Banana Pack MessagePack + more
140. New serialization protocol Banana Pack MessagePack + more mochilo
141. New serialization protocol Banana Pack MessagePack + more mochilo Banana Phone
142. evolutionary (disappointing?)
143. Summary:
144. Ruby C Unix Unix Unix Unix Unix Unix Unix Unix Boring Boring Boring Boring Boring Summary:
145. Use the most reliable tools you know.
146. Challenge yourself to build the simplest thing. Not because it’s easy, but because it works.
147. Innovate where it really ma ers.
148. revolutionary product create a revolutionary backend. not a
150. Q: Does Git scale?
151. Q: Does Git scale? A: Who cares?