Node Interactive US 2016: Real-life node.js troubleshooting

When building a large enough set of services using node.js, there will be a point when you find that your application is suffering from performance or memory issues. When this happens, you have to roll up your sleeves, get your tools and start digging. This talk explains how you can use tools such as ab, flame graphs, heap snapshots and Chrome's memory inspector to find the cause of these. We will go over two real life issues, a CPU bottleneck and a memory leak, we found while building our services at Auth0, and also explain how we fixed them.

Transcript

1. Real-Life Node.js Troubleshooting Damian Schenkelman, Auth0
2. 4 years
3. Usual suspects
4. Memory Leaks
5. –Sebastian Peyrott (a co-worker) “The main cause for leaks in garbage collected languages are unwanted references”
6. Memory Model https://developers.google.com/web/tools/chrome- devtools/memory-problems/memory-101 A B C F E D G GC H I
7. b.d = null A B C F E D G GC H I
8. D not referenced A B C F E D G GC H I
9. D is GCed A B C F E G GC H I
10. Sawtooth
11. Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC
12. Memory Leak A B C F E D G Things we don’t need and are taking up a LOT of space GC More things we don’t need
13. Crashes
14. Memory leak busting
15. Take Control
16. --max-old-space-size is your friend
17. Drain connections
18. Restarts
19. Heap Snapshot
20. v8-profiler https://github.com/node-inspector/v8-profiler var profiler = require('v8-profiler'); var snapshot = profiler.takeSnapshot(); snapshot.export(function(error, result) { fs.writeFileSync('snapshot.json', result); snapshot.delete(); });
21. Demo
26. Mental Picture { 'kinesis.us-west-1.amazonaws.co:443' : } { sockets : } [ …, …, …, …, …, …, …, …,…, TLSSocket ] Something related to logging…
27. Not Easy https://github.com/auth0/kinesis-writable/pull/6/files
28. forever-agent https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 if (this.freeSockets[name] && this.freeSockets[name].length > 0 && !req.useChunkedEncodingByDefault) { var idleSocket = this.freeSockets[name].pop() idleSocket.removeListener('error', idleSocket._onIdleError) delete idleSocket._onIdleError req._reusedSocket = true req.onSocket(idleSocket) } else { this.addRequestNoreuse(req, host, port) }
29. Fixed
30. CPU bottlenecks
31. Auth Service DB Scenario Client
32. Expected response times
33. Actual response times
34. Flame graphs a() b() c() d() h() i() e() f() g()
35. Flame graphs
36. Demo
37. The problem
38. Scale
39. Faster Hash
40. Caching
41. Scale up
42. Multiple Auth Services LB Auth Service Auth Service Auth Service Auth Service
43. BaaS https://github.com/auth0/node-baas // compare bcrypt for req.password to db hash baas.compare(req.body.password, user.passwordHash, (err, success) => { res.send(err || !success ? 401 : 200); });
44. Auto Scaling Auth Service LB BaaS BaaS BaaS BaaS
45. Cost comparison Price / (1M req) #req per sec / vCPU t2-micro $0.36 10.00 t2-medium $0.76 9.50 c4-large $1.53 10.00 c3-8xlarge $1.64 8.88
46. Fail gracefully
47. Cat picture
48. Memory Leaks • Memory terminology: https://developers.google.com/web/tools/ chrome-devtools/memory-problems/memory-101 • Memory leaks in JS: https://auth0.com/blog/four-types-of-leaks- in-your-javascript-code-and-how-to-get-rid-of-them/ • Roots: https://stackoverflow.com/questions/9748358/when- does-the-js-engine-create-a-garbage-collection-root • Increase heap size: https://twitter.com/tjholowaychuk/status/ 480753206301966336 • v8 GC logs speaking to you: https://github.com/joyeecheung/ v8-gc-talk
49. Memory Leaks (2) • Connection draining: https://cloud.google.com/ compute/docs/load-balancing/enabling-connection- draining • master-process: https://github.com/jfromaniello/ master-process • Express connection draining: https://github.com/ expressjs/express/issues/1366 • v8-profiler: https://github.com/node-inspector/v8- profiler
50. Memory Leaks (3) • AWS SDK issues: https://github.com/aws/aws-sdk-js/issues/329, https://github.com/aws/aws-sdk-js/issues/855 • Kinesis event fix: https://github.com/auth0/kinesis-writable/pull/6/files • Forever agent: https://github.com/request/forever-agent/blob/ ece900a6e8dfac734186db3080c00875c0300450/index.js#L70 • Memory profiling for mere mortals: http://thlorenz.com/talks/memory- profiling/book/memory_profiling_for_mere_mortals_/ memory_profiling_for_mere_mortals__0.html • @thlorenz notes on mem profiling: https://github.com/thlorenz/v8- perf/blob/master/memory-profiling.md
51. Performance • http://security.stackexchange.com/a/83382 • http://www.brendangregg.com/FlameGraphs/cpuflamegraphs.html • https://github.com/davidmarkclements/0x • https://github.com/thlorenz/v8-perf/issues/4 • https://github.com/auth0/node-baas • http://docs.aws.amazon.com/AutoScaling/latest/DeveloperGuide/ US_SetUpASLBApp.html • https://gist.github.com/trevnorris/9616784 • https://github.com/dschenkelman/bcrypt-sample
52. Thanks https://github.com/dschenkelman/node-troubleshooting @dschenkelman npm i dschenkelman