Monitoring on a budget

How npm does monitoring & metrics on a budget.

Transcript

1. monitoring on a budget
2. a few animated gifs with the Twelfth Doctor (0 cats)
3. C J Silverio vp of engineering, @ceejbot
4. let's talk npm by the numbers
5. 205 million packages Tuesday 10K requests/sec
6. npm is 25 people 4 of us run the registry
7. when the company was formed 5 people total
8. you outsource many services when you're tiny
9. you pull them back in-house when you succeed
10. success is sometimes a catastrophe
11. npm's scale: runaway success npm's staff: wouldn't this be neat
12. mission: know this on a budget
14. 2 questions: is the registry up? how well is it performing?
15. is the registry up? monitoring
16. how well is it performing? metrics
17. monitoring
18. monitoring == pull ask questions that you know the right answers for
19. Is this host up? Is this cert about to expire? Is the DB replication keeping up?
20. if you get the wrong answer somebody gets paged
21. nagios state of the art in free
23. It's okay. We never look at it. It just triggers Pager Duty.
24. nagios’s virtues: reliability & custom checks
25. goal: never page anybody
26. self-healing checks automate the fix if you can!
27. monitoring == unit tests a ratchet for continuous improvement
28. external monitoring ping services
29. you must monitor but that's just the start
30. monitoring tells you what it doesn't tell you why
31. metrics
32. Q: What's a metric? A: A name + a value + a time.
33. counter: it happened N times gauge: it's Y-sized right now rate: it's happening N times/second timing: it took X milliseconds
34. metrics == push the app gives you numbers
35. emit from a service store in timeseries db query & graph
36. the usual stack statsd ➜ graphite ➜ grafana
39. statsd uses UDP
40. Q: Why not send metrics over UDP? A: You care about receiving them.
41. just try to install graphite
42. for-pay/SAAS services exist but I can't afford them
43. monitoring 400 processes right now 12+ GB of log data a day
44. interlude: when should you pay?
45. convert the £$€ cost into engineer hours/month
46. pay when it's cheaper than investing an engineer (be honest about the cost)
47. numbat was born “How hard can it be?” I said.
49. https://github.com/ numbat-metrics numbat - powered metrics
51. npm’s stack numbat ➜ influxdb ➜ grafana
52. var Emitter = require('numbat-emitter'); var emitter = new Emitter({ uri: 'tcp://localhost:3333', app: 'www', }); process.emit('metric', { name: 'request.latency', value: 30 }); process.emit('metric', { name: 'disk.used.percent', value: 36 }); process.emit('metric', { name: 'login' });
53. so easy to emit a metric that we just do it any time something interesting happens
54. 4000 metrics/sec from the registry
59. metrics ➜ alerts
60. Server handling expected traffic? Latency higher than normal? Error rate higher than usual?
61. metrics comprise a data stream send the stream to more than one place!
62. anomaly detection
63. recap time!
64. your web apps are backed by something
65. what's it up to? how do you know?
66. get data on what your services are up to
67. what: monitoring yes/no questions
68. why: metrics data changing over time
69. next: anomaly detection predictions & trends
70. automate don't require humans
71. npm install -g npm@latest @ceejbot on all the things npm loves you