Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Stabilizing the npm registry
Search
C J Silverio
February 11, 2015
Programming
2
340
Stabilizing the npm registry
How npm went from a car fire to a very boring service with a very boring uptime.
C J Silverio
February 11, 2015
Tweet
Share
More Decks by C J Silverio
See All by C J Silverio
The economics of package management
ceejbot
4
1.5k
The future of (javascript) modules (in node)
ceejbot
1
270
Keeping JavaScript safe
ceejbot
3
400
ceej's how to solve it
ceejbot
6
760
work-life balance at npm
ceejbot
5
780
hash functions and you!
ceejbot
2
340
The accidental noder
ceejbot
2
140
Design Patterns & Modularity in the npm Registry
ceejbot
3
180
Monitoring on a budget
ceejbot
2
280
Other Decks in Programming
See All in Programming
のびしろを広げる巻き込まれ力:偶然を活かすキャリアの作り方/oso2024
takahashiikki
1
410
カスタムしながら理解するGraphQL Connection
yanagii
1
1.2k
CSC509 Lecture 08
javiergs
PRO
0
110
アジャイルを支えるテストアーキテクチャ設計/Test Architecting for Agile
goyoki
7
2.8k
レガシーな Android アプリのリアーキテクチャ戦略
oidy
1
170
Modern Angular: Renovation for Your Applications
manfredsteyer
PRO
0
200
Content Security Policy入門 セキュリティ設定と 違反レポートのはじめ方 / Introduction to Content Security Policy Getting Started with Security Configuration and Violation Reporting
uskey512
1
420
Streams APIとTCPフロー制御 / Web Streams API and TCP flow control
tasshi
1
290
Universal Linksの実装方法と陥りがちな罠
kaitokudou
1
220
Googleのテストサイズを活用したテスト環境の構築
toms74209200
0
270
Jakarta Concurrencyによる並行処理プログラミングの始め方 (JJUG CCC 2024 Fall)
tnagao7
1
230
/←このスケジュール表に立ち向かう フロントエンド開発戦略 / A front-end development strategy to tackle a single-slash schedule.
nrslib
1
590
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
305
110k
Optimising Largest Contentful Paint
csswizardry
33
2.9k
Why You Should Never Use an ORM
jnunemaker
PRO
53
9k
Adopting Sorbet at Scale
ufuk
73
9k
Imperfection Machines: The Place of Print at Facebook
scottboms
264
13k
The Power of CSS Pseudo Elements
geoffreycrofte
72
5.3k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
41
2.1k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
28
7.9k
Embracing the Ebb and Flow
colly
84
4.4k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
131
33k
Practical Tips for Bootstrapping Information Extraction Pipelines
honnibal
PRO
9
680
A Tale of Four Properties
chriscoyier
156
23k
Transcript
stabilizing the registry
C J Silverio director of engineering, npm @ceejbot
This is the story of a plucky package registry named
npm
scaling problem manifesting itself as a stability problem
"scaling" capacity to meet growing demands
"At scale" huge demand & lots of data
"stability" not falling over under normal demand
What's normal demand?
129K packages 239 GB package tarballs 40 million pkg dls/day
1500 req/sec, peak 3200
"Legacy" Anything you've put into production
this is the story of a legacy system becoming more
flexible
None
January 2013 20K packages .5 million dls/day
Oct 2013 44K packages 108 million dls/month 3.6 million dls/day
None
our plucky little registry had to change
step 1: CDN Put Fastly.com in front of the registry
cache rules everything around me
step 2: tarballs get them out of couchdb
tarballs are huge! couch runs better without them base64 decoding
is work.
None
January 2014 60K packages 6+ million dls/day
step 3: visibility are things going wrong? what's going wrong?
reactive monitoring monitor deeply fix things quickly
proactive monitoring self-healing (also things don't break)
monitoring is unit testing Add monitoring after every outage
visibility is a prerequisite but not a solution
act on what monitoring and metrics reveal
step 4: redundancy several CouchDBs! reads, writes, & replication
fewer responsibilities for each piece isolates errors
step 5: automation ansible no server is special
June 2014 Superficially similar.
June 2014 80K packages 10 million dls/day
step 6: simplification now that it's not on fire we
can modify at leisure
None
Nov 2014 105K packages 28 million dls/day peak
50/50 AWS region split no AWS-specific magic Ubuntu 14.04 Trusty
Fastly: geoloc + varnish haproxy + CouchDB nginx + a
filesystem
where's the node?
registry 2 electric boogaloo with 500% more node
None
haproxy + node services couchdb ➜ postgres redis for caching
nginx + filesystem
more complicated more flexible & redundant more scaling dials to
turn
excited about postgres ad-hoc queries are fun
scaling node is exactly like scaling everything else
Understand system get visibility cool down hot spots add redundancy
npm client <3 npm install -g npm@latest
npm loves you