PDX 2017 - Roy Rapoport

Transcript

1. Monitoring Monitoring Systems at Roy Rapoport - @royrapoport May 22, 2017
2. The Hero’s Journey
3. An ordinary person A software engineer
4. Receives a call Lets build a monitoring system!
5. is dubious These specs are silly and you’re silly
6. is persuaded nonetheless Sure, I’d love to learn Scala!
7. vanquishes enemies Product Managers Engineering Managers Opinionated Coworkers
8. undergoes the ordeal Damn, this won’t scale. Also, we probably shouldn’t have coded a JS interpreter in Erlang.
9. returns, successful and transformed This will scale for at least a month. Ship it!
10. The Beginning
11. Real-Time Operational Insight
13. Monitoring ain’t Alerting
14. Alerting
15. Alert Action Server Stream Processing Atlas SQS Hot Checker Cold Checker AWS Email Page …
16. Monitoring
17. Preamble
18. 6H 3D 18D Publish MR Query S3 Anatomy of an Atlas Deployment Group (ADG) m etrics
19. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy
20. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG ADG
21. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG ADG iep ADG
 iep ADG
 iep ADG

22. Server Proxy us-west-2 Server Proxy us-east-1 eu-west-1 Server Proxy ADG ADG iep ADG
 iep ADG
 iep ADG
 Critical Critical Critical
23. Tradeoffs: Fail loud Lower TTD
24. “This morning around 0504 US/Pacific, the active primary region hosting Atlas metrics data began experiencing issues and a number of alerts were triggered across the board.“ February 9, 2017
26. Managing Cost
27. Cost Doesn’t Matter
31. “Everything Counts in Large Amounts”
32. “Everything Counts in Large Amounts” - Depeche Mode
33. Cardinal Sins
34. What’s a Metric? • name = cpu • nf.region = us-east-1 • nf.zone = us-east-1a • nf.app = api • nf.cluster = api-main • nf.asg = api-main-v103 • nf.node = i-103efbdd • device.operationalName = roku • isSupplementalVideo = false • OnComplete = operation • …
35. OOPS
36. Fun Metric Dimensions
 A True Story • IP address (~ 232) • Account ID (~109) • Customer ID (~109) • Execution time, ms (~30K)
41. “The graph on the wall tells the story of it all” - Depeche Mode
43. What are you (not) talking about?
44. The latest thing
47. Meet Swaroop
48. The End
49. The Beginning
50. ?
52. Attributions • https://www.flickr.com/photos/kalleboo/2240042641/ • http://imgur.com/gallery/QCrR73q • https://www.flickr.com/photos/35028096@N04/6119259650 • https://www.flickr.com/photos/86639298@N02/9420223184 • http://paristext2015.com/2015/06/1-5-degrees-celsius-or-2-degrees-maybe-its-turtles-all- the-way-down/ • https://www.flickr.com/photos/60969081@N00/8292545493 • By Photo courtesy of National Nuclear Security Administration / Nevada Site Office - This image is available from the National Nuclear Security Administration Nevada Site Office Photo Library under number XX-34.This tag does not indicate the copyright status of the attached work. A normal copyright tag is still required., Public Domain, https:// commons.wikimedia.org/w/index.php?curid=190949 • http://www.animal-space.net/2010/12/16-funny-pictures-of-shocked-cats.html • https://www.flickr.com/photos/10459174@N03/34563622796/