Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
340
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.5k
The future of (javascript) modules (in node)
ceejbot
1
270
Keeping JavaScript safe
ceejbot
3
400
ceej's how to solve it
ceejbot
6
750
work-life balance at npm
ceejbot
5
770
hash functions and you!
ceejbot
2
340
The accidental noder
ceejbot
2
140
Design Patterns & Modularity in the npm Registry
ceejbot
3
180
Monitoring on a budget
ceejbot
2
280
Other Decks in Programming
See All in Programming
Rechartsで楽にゴリゴリにカスタマイズする!
10tera
1
170
事業フェーズの変化に対応する 開発生産性向上のゼロイチ
masaygggg
0
210
Kotlin 2.0 and Beyond
antonarhipov
2
150
Prompt Cachingは本当に効果的なのか検証してみた.pdf
ttnyt8701
0
530
エラーレスポンス設計から考える、0→1開発におけるGraphQLへの向き合い方
bicstone
5
1.5k
GraphQL あるいは React における自律的なデータ取得について
quramy
11
3k
AndroidアプリのUIバリエーションをあの手この手で確認する / Check UI variations of Android apps by various means
tkmnzm
1
190
LangChainの現在とv0.3にむけて
os1ma
4
940
Android開発以外のAndroid開発経験の活かしどころ
konifar
2
1k
Using Livebook to build and deploy internal tools @ ElixirConf 2024
hugobarauna
0
250
Pythonで改めて考える「クラス(class)」の使いどころ
os1ma
4
900
Swiftコードバトル必勝法
toshi0383
0
170
Featured
See All Featured
What's in a price? How to price your products and services
michaelherold
242
11k
Creatively Recalculating Your Daily Design Routine
revolveconf
215
12k
Building a Scalable Design System with Sketch
lauravandoore
459
32k
How To Stay Up To Date on Web Technology
chriscoyier
786
250k
The Cult of Friendly URLs
andyhume
76
6k
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
28
1.6k
What's new in Ruby 2.0
geeforr
340
31k
Atom: Resistance is Futile
akmur
261
25k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
166
48k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
24
610
Side Projects
sachag
451
42k
Art, The Web, and Tiny UX
lynnandtonic
294
20k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you