Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
380
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.6k
The future of (javascript) modules (in node)
ceejbot
1
290
Keeping JavaScript safe
ceejbot
3
450
ceej's how to solve it
ceejbot
6
760
work-life balance at npm
ceejbot
5
790
hash functions and you!
ceejbot
2
350
The accidental noder
ceejbot
2
150
Design Patterns & Modularity in the npm Registry
ceejbot
3
190
Monitoring on a budget
ceejbot
2
290
Other Decks in Programming
See All in Programming
株式会社 Sun terras カンパニーデック
sunterras
0
170
ててべんす独演会〜Flowの全てを語ります〜
tbsten
1
220
defer f()とdefer fの挙動を 誤解していた話
kogamochiduki
2
160
dynamic!
moro
9
4.6k
Introducing ReActionView: A new ActionView-Compatible ERB Engine @ Kaigi on Rails 2025, Tokyo, Japan
marcoroth
3
690
CSC305 Lecture 01
javiergs
PRO
1
380
プロダクト開発をAI 1stに変革する〜SaaS is dead時代で生き残るために〜 / AI 1st Product Development
kobakei
0
440
Conquering Massive Traffic Spikes in Ruby Applications with Pitchfork
riseshia
0
130
CSC509 Lecture 03
javiergs
PRO
0
320
iOSエンジニア向けの英語学習アプリを作る!
yukawashouhei
0
160
Reduxモダナイズ 〜コードのモダン化を通して、将来のライブラリ移行に備える〜
pvcresin
2
630
なぜあの開発者はDevRelに伴走し続けるのか / Why Does That Developer Keep Running Alongside DevRel?
nrslib
1
160
Featured
See All Featured
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
15
1.7k
How to Ace a Technical Interview
jacobian
280
23k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
162
15k
Build The Right Thing And Hit Your Dates
maggiecrowley
37
2.9k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
51k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Large-scale JavaScript Application Architecture
addyosmani
513
110k
Thoughts on Productivity
jonyablonski
70
4.8k
Speed Design
sergeychernyshev
32
1.1k
Learning to Love Humans: Emotional Interface Design
aarron
274
40k
StorybookのUI Testing Handbookを読んだ
zakiyama
31
6.2k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you