JS-GIT

@isntitvacant NODEPDX 2013 Git JS Thursday, May 16, 13

@isntitvacant NODEPDX 2013 Thursday, May 16, 13

@isntitvacant NODEPDX 2013 HI.my name is chris. I work at
. (it’s fun) (As a JS engineer) (We're hiring) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 wHY GIT? Thursday, May 16, 13

@isntitvacant NODEPDX 2013 Made a GIT/HG site in 2010 with
a friend “repocracy” very nearly immediately trolled wHY? Thursday, May 16, 13

@isntitvacant NODEPDX 2013 wHY? wHY? Thursday, May 16, 13

@isntitvacant NODEPDX 2013 wHY? wanted to limit repo size before
cloning hence, “tempisfugit” github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

@isntitvacant NODEPDX 2013 wHY? tempisfugit was awful, just awful no
zlib in node yet heavily tied to fs api monolithic And, of course... github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

@isntitvacant NODEPDX 2013 fs.stat(baseFolder, function(err, stats) { err && exit(err,
null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? Thursday, May 16, 13

null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit Thursday, May 16, 13

null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(baseFol fs.stat(objectsFolder, function(err, stats) { err && exit(err, null) || (function() { !stats.isDirectory() && exit(errors.NOTAREPO(o fs.readFile(indexFile, function(err, indexDa err && exit(err, null) || (function() { fs.readFile(headFile, function(err, head err && exit(err, null) || (function() var odbObject = odb.ODB.init(); exit(null, new Repository(odbObject, 'repository':baseFolder, 'index':indexFile, 'head':headFile, 'objects':objectsFolder, wHY? github.com/chrisdickinson/tempisfugit MISTAKES WERE MADE Thursday, May 16, 13

@isntitvacant NODEPDX 2013 wHY? 2-3 months ago js-git was funded
wanted to “back with code” also to apply experience working on voxel.js Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT is git? Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS Thursday, May 16,
13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TAGS Thursday, May
16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS COMMITS TAGS Thursday,
May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 4OBJECTS TREES COMMITS TAGS
Thursday, May 16, 13

BLOBS Thursday, May 16, 13

BLOBS object 171d82736413d1f781a16777bb67c270de9cc084 type commit tag v0.0.1 tagger Chris Dickinson \ <[email protected]> \ Sun Mar 24 00:10:34 2013 -0700 0.0.1 Thursday, May 16, 13

BLOBS tree 27bae442f94a74dd7b070d8a37aa666665e6e123 parent dfa6cf3ebb37c53825035ca67482d3fc4c17e029 author Chris Dickinson \ <[email protected]> \ 1364109034 -0700 committer Chris Dickinson \ <[email protected]> \ 1364109034 -0700 0.0.1 Thursday, May 16, 13

BLOBS 100644 blob f9fe4b03cd774c735d4eda711fbadc199c1f77af README.md 100644 blob 3ec1605cfba1dee1a31c76ccbefcd84b7a285d28 examples.js 100644 blob b5e785955ecd0329f6b771f32a7548f1a65e210a index.js 100644 blob 04e7de77ff4351c08fd37aa2079fd6f23a19e9c0 package.json Thursday, May 16, 13

BLOBS { "name": "git-fs-repo", "version": "0.0.1", "description": "filesystem backed git repository", "main": "index.js", "scripts": { "test": "node test.js" }, ... Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES Thursday, May
16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES LOOSE Thursday,
May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES PACKFILES LOOSE

.git/objects/{2 hex}/{38 hex} “commiT”, “BLOB”, “TAG”, “TREE” literal DECIMAL UNCOMPRESSED SIZE DEFLATED OBJECT DATA Thursday, May 16, 13

.git/objects/pack/{pack-hash}.idx .git/objects/pack/{pack-hash}.pack IDX =”index” random access for git objects in pack this is where all the deltas live more on packfiles later! Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 2OBJECT STORES var log
= require('git-walk-refs') var repo = require('git-fs-repo') repo('path/to/.git', function(err, git) { var hashes = git.refs().map(function(ref) { return ref.hash }) log(git.find, hashes) .on('data', console.log) }) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT Thursday, May 16,
13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES Thursday, May
16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 4 bytes:
“PACK” 4 bytes: version 4 bytes: # of objs HEADER Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES 1 bit:
continuation 3 bits: obj type 4 + N bits: obj size DEFLATED OBJ DATA PACKFILES Object Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 20
bytes: checksum checksum Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... 4
valid types + 2 delta types “offset” and “reference” delta deltas are xdelta format reference has a hash, offset is an offset reference to a previous obj Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT PACKFILES ... deltas
may reference other deltas the packing heuristics are neat ... though totally optional Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var fs
= require('fs') var list = require('git-list-pack') var objectify = require('git-objectify-pack') var objects = {} var find = function(hash, ready) { return ready(null, objects[hash]) } fs.createReadStream('.git/objects/XXX.pack') .pipe(list()) .pipe(objectify(find)) .on('data', function(obj) { objects[obj.hash] = obj }) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn
= net.connect({host: 'github.com', port: 9418}) var fetch = require('git-fetch-pack') var transport = require('git-transport-protocol') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/(master)/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(yourFavoriteKVStore()) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 WHAT? GIT IS 1TRANSPORT ... var conn
= shoe.connect('/git') var fetch = require('git-fetch-pack') var client client = fetch( 'git://github.com/chrisdickinson/plate.git' , function want(ref, ready) { return ready(/master/.test(ref.name)) }) ) client .pipe(transport(conn)) .pipe(client) client.pack .pipe(list()) .pipe(objectify(find)) .pipe(levelJS()) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 Lessons. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

@isntitvacant NODEPDX 2013 write modules lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16,
13

@isntitvacant NODEPDX 2013 write modules (LOTS OF small) lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO

@isntitvacant NODEPDX 2013 lessons. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. github.com/CHRISDICKINSON/JS-GIT-DEMO Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. easier to test Thursday, May 16,
13

@isntitvacant NODEPDX 2013 lessons. easier to test easier to write

easier to document Thursday, May 16, 13

easier to document easier to change later Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. USE Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. USE github.com/substack/node-browserify Thursday, May 16, 13

@isntitvacant NODEPDX 2013 great build tool (with standalone) lessons. Thursday,
May 16, 13

@isntitvacant NODEPDX 2013 great build tool (with standalone) great for
shimming lessons. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 great build tool (with standalone) great for
shimming great tooling (source maps, beefy, testling ci) lessons. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. put apis in terms of Thursday,
May 16, 13

@isntitvacant NODEPDX 2013 lessons. put apis in terms of streams

@isntitvacant NODEPDX 2013 lessons. pluggable api Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. pluggable api don’t need the entire
thing working to test Thursday, May 16, 13

thing working to test don’t need all of the data at once to process Thursday, May 16, 13

thing working to test don’t need all of the data at once to process use through! Thursday, May 16, 13

thing working to test don’t need all of the data at once to process use through! (and min-streams underneath) Thursday, May 16, 13

@isntitvacant NODEPDX 2013 thanks. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson Thursday, May 16, 13

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant Thursday, May
16, 13

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git
/ freenode Thursday, May 16, 13

@isntitvacant NODEPDX 2013 thanks. github: chrisdickinson twitter: @isntitvacant irc: #js-git
/ freenode and thanks to creationix, dominictarr, danlucraft, maksimlin, and others! Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 lessons. ZLIB Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s

@isntitvacant NODEPDX 2013 ISSUES. ZLIB there’s a shim + node’s
BUT you can’t get the unused bytes from inflate Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate Thursday, May 16,
13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me
sad Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. ZLIB rewrote inflate this makes me
sad but also happy! was fun. github.com/CHRISDICKINSON/inflate Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,
13

@isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere bu ff
ers Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. browserify’s shim works everywhere BUT typed
arrays are better in the browser github.com/CHRISDICKINSON/bops bu ff ers Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. bu ff ers Thursday, May 16,
13

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops bu ff ers Thursday, May
16, 13

@isntitvacant NODEPDX 2013 ISSUES. github.com/CHRISDICKINSON/bops use a different primitive module
bu ff ers Thursday, May 16, 13

“bops” - buffer ops bu ff ers Thursday, May 16, 13

“bops” - buffer ops use typed arrays in browser, buffers elsewhere bu ff ers Thursday, May 16, 13

@isntitvacant NODEPDX 2013 small modules ISSUES. Thursday, May 16, 13

@isntitvacant NODEPDX 2013 small modules ISSUES. using a different primitive
is difficult Thursday, May 16, 13

is difficult requires making all modules dep on it Thursday, May 16, 13

is difficult requires making all modules dep on it and bumping those deps Thursday, May 16, 13

is difficult requires making all modules dep on it and bumping those deps voxel.js ran into this with three.js Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. small modules Thursday, May 16, 13

@isntitvacant NODEPDX 2013 ISSUES. not insurmountable, though small modules Thursday,
May 16, 13

JS-GIT

JS-GIT

More Decks by Chris Dickinson

Other Decks in Programming

Featured

Transcript