Node.js: scalability tips

You finally built that amazing start-up idea you had in mind for years and you did it using Node.js! That's Great! You just launched it on Hacker News and you are very happy and proud... but now more and more people are using it and you start to have a nasty fear that Node.js won't scale because you now... it's single-threaded! Is your project doomed now? Do you have to invest your time on rewriting it in something like C++ or maybe Rust or even Go? You'd rather invest your time on adding valuable features for your users rather than learning a new language and rewriting everything from scratch, but what if nothing works anymore? And... by the way, what the heck "single-threaded" really means?! Fear no more, dear fellow developer! In this talk, we will discuss the architecture of Node.js going through its strengths and its weaknesses. We will then talk about scalability and I will share some valuable tips and tricks to make your Node.js app scale! Spoiler alert: you probably won't need Go or Rust :)

Transcript

1. Node.js Scalability Tips Luciano Mammino ( ) @loige 2020-07-09 loige.link/node-scale 1
2. Hello, I am Luciano! Principal Software Engineer at FabFitFun Blog: Twitter: GitHub: loige.co @loige @lmammino nodejsdp.link/3rd 2
3. @loige Get the slides! loige.link/node-scale 3
4. Node.js + Scalability? @loige 4
5. @loige 5
6. @loige 6
7. @loige 6
8. @loige 6
9. @loige 6
10. @loige 7
11. @loige 7
12. @loige 7
13. @loige 7
14. @loige 7
15. "Scalability is the property of a system to handle a growing amount of work by adding resources to the system" — Wikipedia @loige 8
16. "A service is said to be scalable if when we increase the resources in a system, it results in increased performance in a manner proportional to resources added" — Werner Vogels @loige 9
17. Tip 1. Establish a baseline @loige 10
18. /?data=Hello%20Shift https://repl.it/@lmammino/QRGen @loige 11
19. const { createServer } = require('http') const { URL } = require('url') const QRCode = require('qrcode') createServer(function handler (req, res) { const url = new URL(req.url, 'http://localhost:8080') const data = url.searchParams.get('data') if (!data) { res.writeHead(400) // bad request return res.end() } res.writeHead(200, { 'Content-Type': 'image/png' }) QRCode.toFileStream(res, data, { width: 300 }) }) .listen(8080) @loige 12
20. autocannon -c 200 --on-port / -- node server.js wrk node server.js& wrk -t8 -c200 -d10s http://localhost:8080/ @loige 13
21. autocannon -c 200 --on-port /?data=Hello%20Shift -- node server.js Running 10s test @ http://localhost:8080/?data=Hello%20Shift 200 connections ┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬─────────┬────────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼─────────┼────────────┤ │ Latency │ 1899 ms │ 1951 ms │ 2053 ms │ 2054 ms │ 1964.92 ms │ 99.9 ms │ 3364.03 ms │ └─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴─────────┴────────────┘ ┌───────────┬─────┬──────┬─────────┬────────┬────────┬────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤ │ Req/Sec │ 0 │ 0 │ 30 │ 199 │ 99.5 │ 94.27 │ 30 │ ├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤ │ Bytes/Sec │ 0 B │ 0 B │ 50.7 kB │ 336 kB │ 168 kB │ 159 kB │ 50.7 kB │ └───────────┴─────┴──────┴─────────┴────────┴────────┴────────┴─────────┘ Req/Bytes counts sampled once per second. 995 requests in 10.08s, 1.68 MB read @loige 14
22. autocannon -c 200 --on-port /?data=Hello%20Shift -- node server.js Running 10s test @ http://localhost:8080/?data=Hello%20Shift 200 connections ┌─────────┬─────────┬─────────┬─────────┬─────────┬────────────┬─────────┬────────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼─────────┼─────────┼─────────┼─────────┼────────────┼─────────┼────────────┤ │ Latency │ 1899 ms │ 1951 ms │ 2053 ms │ 2054 ms │ 1964.92 ms │ 99.9 ms │ 3364.03 ms │ └─────────┴─────────┴─────────┴─────────┴─────────┴────────────┴─────────┴────────────┘ ┌───────────┬─────┬──────┬─────────┬────────┬────────┬────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤ │ Req/Sec │ 0 │ 0 │ 30 │ 199 │ 99.5 │ 94.27 │ 30 │ ├───────────┼─────┼──────┼─────────┼────────┼────────┼────────┼─────────┤ │ Bytes/Sec │ 0 B │ 0 B │ 50.7 kB │ 336 kB │ 168 kB │ 159 kB │ 50.7 kB │ └───────────┴─────┴──────┴─────────┴────────┴────────┴────────┴─────────┘ Req/Bytes counts sampled once per second. 995 requests in 10.08s, 1.68 MB read @loige 14
23. ⛅ Tip 1-bis Also, find out your ceiling @loige 15
24. const { createServer } = require('http') createServer((req, res) => { if (req.method === 'GET' && req.url === '/') { res.writeHead(200, { 'Content-Type': 'text/plain' }) res.end('Hello World\n') } else { res.statusCode = 404 res.end() } }) .listen(8080) @loige 16
25. autocannon -c 200 --on-port / -- node server.js Running 10s test @ http://localhost:8080/ 200 connections ┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────────┤ │ Latency │ 3 ms │ 5 ms │ 11 ms │ 14 ms │ 5.51 ms │ 2.71 ms │ 80.63 ms │ └─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────────┘ ┌───────────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤ │ Req/Sec │ 21087 │ 21087 │ 34623 │ 35487 │ 33258.4 │ 4107.01 │ 21077 │ ├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤ │ Bytes/Sec │ 3.29 MB │ 3.29 MB │ 5.4 MB │ 5.54 MB │ 5.19 MB │ 641 kB │ 3.29 MB │ └───────────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┴─────────┘ Req/Bytes counts sampled once per second. 333k requests in 10.1s, 51.9 MB read @loige 17
26. autocannon -c 200 --on-port / -- node server.js Running 10s test @ http://localhost:8080/ 200 connections ┌─────────┬──────┬──────┬───────┬───────┬─────────┬─────────┬──────────┐ │ Stat │ 2.5% │ 50% │ 97.5% │ 99% │ Avg │ Stdev │ Max │ ├─────────┼──────┼──────┼───────┼───────┼─────────┼─────────┼──────────┤ │ Latency │ 3 ms │ 5 ms │ 11 ms │ 14 ms │ 5.51 ms │ 2.71 ms │ 80.63 ms │ └─────────┴──────┴──────┴───────┴───────┴─────────┴─────────┴──────────┘ ┌───────────┬─────────┬─────────┬────────┬─────────┬─────────┬─────────┬─────────┐ │ Stat │ 1% │ 2.5% │ 50% │ 97.5% │ Avg │ Stdev │ Min │ ├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤ │ Req/Sec │ 21087 │ 21087 │ 34623 │ 35487 │ 33258.4 │ 4107.01 │ 21077 │ ├───────────┼─────────┼─────────┼────────┼─────────┼─────────┼─────────┼─────────┤ │ Bytes/Sec │ 3.29 MB │ 3.29 MB │ 5.4 MB │ 5.54 MB │ 5.19 MB │ 641 kB │ 3.29 MB │ └───────────┴─────────┴─────────┴────────┴─────────┴─────────┴─────────┴─────────┘ Req/Bytes counts sampled once per second. 333k requests in 10.1s, 51.9 MB read @loige 17
27. Tip 2. Find your bottleneck @loige 18
28. Clinic.js clinicjs.org @loige 19
29. clinic doctor --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js @loige 20
30. clinic flame --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js @loige 21
31. clinic bubble --autocannon [ -c 200 '/?data=Hello%20Shift' ] -- node server.js @loige 22
32. Tip 3. Understand your goals @loige 23
33. What do we optimize for? Throughput? Memory? Latency? @loige 24
34. Tip 4. Always "observe" @loige 25
35. I mean, in production! Logs - Metrics - Traces @loige 26
36. Tip 5. Scale your architecture @loige 27
37. Performance != Scalability @loige 28
38. How can we scale a system by adding resources? @loige 29
39. The " " Scale Cube x-axis cloning z-axis partitioning y-axis functional decomposition @loige 30
40. The " " Scale Cube x-axis cloning z-axis partitioning y-axis functional decomposition @loige 30
41. Cloning Reverse proxy 31 Inside the same server Load Balancer Using multiple server @loige
42. The module cluster Master process Worker process Worker process Worker process 32 @loige
43. The module cluster Master process Worker process Worker process Worker process 32 @loige
44. The module cluster Master process Worker process Worker process Worker process 32 @loige
45. The module cluster Master process Worker process Worker process Worker process 32 @loige
46. The module cluster Master process Worker process Worker process Worker process 32 @loige
47. The module cluster Master process Worker process Worker process Worker process 32 @loige
48. The module cluster Master process Worker process Worker process Worker process 32 @loige
49. const cluster = require('cluster') const numCPUs = require('os').cpus().length if (cluster.isMaster) { // Fork workers for (let i = 0; i < numCPUs; i++) { cluster.fork() } } else { // Worker code require('./server.js') } @loige 33
50. const cluster = require('cluster') const numCPUs = require('os').cpus().length if (cluster.isMaster) { // Fork workers for (let i = 0; i < numCPUs; i++) { cluster.fork() } } else { // Worker code require('./server.js') } 3-4x req/sec (8 core) @loige 33
51. You could also use Check out ! Worker Threads piscina @loige 34
52. Cloning is the easiest strategy to scale a service... ... as long as your application is Stateless @loige 35
53. API Gateway Functional decomposition a.k.a. "Micro-services" 36 /products /cart cart DB products DB @loige
54. API Gateway Functional decomposition a.k.a. "Micro-services" 37 /products /cart Functional decomposition can also be combined with cloning! cart DB products DB @loige
55. Node.js is great for microservices @loige 38
56. Microservices can also help with scaling the organisation! @loige 39
57. Microservices add complexity Observability Deployments Versioning Integration @loige 40
58. Partitioning Service and Data Partitioning along Customer Boundaries Shard partitioning /products/[A-L]/ /products/[M-Z]/ DB 2 41 DB 1 @loige
59. Partitioning is generally used to scale databases and SaaS software geographically @loige 42
60. Summary @loige 43
61. Summary Establish a baseline @loige 43
62. Summary Establish a baseline Find your bottleneck @loige 43
63. Summary Establish a baseline Find your bottleneck Understand your goals @loige 43
64. Summary Establish a baseline Find your bottleneck Understand your goals Always "observe" @loige 43
65. Summary Establish a baseline Find your bottleneck Understand your goals Always "observe" Scale your architecture (cloning, decomposition & partitioning) @loige 43
66. Thank you! Special thanks to , , , , , , , , , , , , , , , , , , , , , Icons and SVGs by @StefanoAbalsamo @matteocollina @dagonzago @NullishCoalesce @DublinSvelte @KViglucci @gjohnson391 @lucamaraschi @laurekamalandua @giltayar @mrm8488 @adrirai @harafise @EugeneWare @Jauny @tlivings @michaelcfine @leojino @shahidontech @Lordoomer @zsadigov @dottorblaster freepik.com loige.link/node-scale @loige 44