Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
340
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.5k
The future of (javascript) modules (in node)
ceejbot
1
270
Keeping JavaScript safe
ceejbot
3
400
ceej's how to solve it
ceejbot
6
760
work-life balance at npm
ceejbot
5
780
hash functions and you!
ceejbot
2
340
The accidental noder
ceejbot
2
140
Design Patterns & Modularity in the npm Registry
ceejbot
3
180
Monitoring on a budget
ceejbot
2
280
Other Decks in Programming
See All in Programming
Laravel や Symfony で手っ取り早く OpenAPI のドキュメントを作成する
azuki
2
120
聞き手から登壇者へ: RubyKaigi2024 LTでの初挑戦が 教えてくれた、可能性の星
mikik0
1
130
Duckdb-Wasmでローカルダッシュボードを作ってみた
nkforwork
0
130
距離関数を極める! / SESSIONS 2024
gam0022
0
290
シェーダーで魅せるMapLibreの動的ラスタータイル
satoshi7190
1
480
イベント駆動で成長して委員会
happymana
1
330
ローコードSaaSのUXを向上させるためのTypeScript
taro28
1
630
subpath importsで始めるモック生活
10tera
0
310
카카오페이는 어떻게 수천만 결제를 처리할까? 우아한 결제 분산락 노하우
kakao
PRO
0
110
C++でシェーダを書く
fadis
6
4.1k
PHP でアセンブリ言語のように書く技術
memory1994
PRO
1
170
どうして僕の作ったクラスが手続き型と言われなきゃいけないんですか
akikogoto
1
120
Featured
See All Featured
It's Worth the Effort
3n
183
27k
Art, The Web, and Tiny UX
lynnandtonic
297
20k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
370
Designing for humans not robots
tammielis
250
25k
A Philosophy of Restraint
colly
203
16k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
169
50k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
10 Git Anti Patterns You Should be Aware of
lemiorhan
655
59k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
10
720
KATA
mclloyd
29
14k
Gamification - CAS2011
davidbonilla
80
5k
Code Review Best Practice
trishagee
64
17k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you