[2016.05 Meetup #2][TALK #3] Hassy Veldstra - Chaos Llama – a modern take on the Netflix Chaos Monkey

Hassy Veldstra's talk slides: Chaos Llama – a modern take on the Netflix Chaos Monkey

All rights belong to Hassy Veldstra.

1. Chaos Llama – a modern take on the Netflix Chaos Monkey Hassy Veldstra <h@veldstra.org> @hveldstra
2. # whoami • Backend Node.js & DevOps engineer
3. # whoami • Backend Node.js & DevOps engineer • Interested in reliability and performance
4. # whoami • Backend Node.js & DevOps engineer • Interested in reliability and performance • artillery.io
6. Chaos Llama • Like the Chaos Monkey • That runs on AWS Lambda
7. Why would you do that?
8. Chaos Engineering
9. Chaos Engineering • .
10. Chaos Engineering • . • “The only way to prepare for system failures is to simulate them”
11. Chaos Engineering • . • “The only way to prepare for system failures is to simulate them” • .
12. Chaos Engineering • Modern systems are complex
13. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex
14. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses:
15. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses: • Improper fallback settings when a subsystem is down
16. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses: • Improper fallback settings when a subsystem is down • Retry storms (thundering herd) from improperly tuned timeouts
17. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses : • Improper fallback settings when a subsystem is down • Retry storms (thundering herd) from improperly tuned timeouts • Cascading failures when a SPOF fails
18. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses: • Improper fallback settings when a subsystem is down • Retry storms (thundering herd) from improperly tuned timeouts • Cascading failures when a SPOF fails • Improperly tuned connection pools
19. Chaos Engineering • Modern systems are complex • Consequence: failure modes are complex • Common weaknesses: • Improper fallback settings when a subsystem is down • Retry storms (thundering herd) from improperly tuned timeouts • Cascading failures when a SPOF fails • Improperly tuned connection pools • Many more
20. Chaos Engineering • The only way to build a resilient system is to stress test it with failure and fix weaknesses
21. Chaos Engineering
22. Chaos Engineering 1. Define the “steady state”
23. Chaos Engineering 1. Define the “steady state” 1. P95 response time
24. Chaos Engineering 1. Define the “steady state” 1. P95 response time 2. Error rates (e.g. HTTP 500)
25. Chaos Engineering 1. Define the “steady state” 1. P95 response time 2. Error rates (e.g. HTTP 500) 3. E-commerce app: completed orders
26. Chaos Engineering 1. Define the “steady state” 1. P95 response time 2. Error rates (e.g. HTTP 500) 3. E-commerce app: completed orders 2. Inject failure into the system
27. Chaos Engineering 1. Define the “steady state” 1. P95 response time 2. Error rates (e.g. HTTP 500) 3. E-commerce app: completed orders 2. Inject failure into the system 3. Monitor changes in the steady state
28. Chaos Engineering 1. Define the “steady state” 1. P95 response time 2. Error rates (e.g. HTTP 500) 3. E-commerce app: completed orders 2. Inject failure into the system 3. Monitor changes in the steady state 4. Fix weaknesses
29. Advanced Chaos Engineering
30. Advanced Chaos Engineering 1. Do it in production
31. Advanced Chaos Engineering 1. Do it in production 2. Automate these chaos experiments to run continuously
32. http://principlesofchaos.org/
33. Not just for Netflix
34. Chaos Monkey
35. Chaos Monkey • Released in 2012
36. Chaos Monkey • Released in 2012 • Popular open-source project
37. Chaos Monkey - Downsides • Large codebase (thousands of lines of Java)
38. Chaos Monkey - Downsides • Large codebase (thousands of lines of Java) • Requires an EC2 instance
39. Chaos Monkey - Downsides • Large codebase (thousands of lines of Java) • Requires an EC2 instance • Lots of config to get going
40. Chaos Monkey - Downsides • Large codebase (thousands of lines of Java) • Requires an EC2 instance • Lots of config to get going • Quick start guide is 16 pages
43. Llama vs Monkey • Serverless vs Need an EC2 instance
44. Llama vs Monkey • Serverless vs Need an EC2 instance • Easy to set up vs Lots of config
45. Llama vs Monkey • Serverless vs Need an EC2 instance • Easy to set up vs Lots of config • Tiny (400 LoC) vs Huge (thousands)
46. AWS Lambda 101
47. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc)
49. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code
50. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”)
51. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”) • The piece of code is a “lambda function”
52. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”) • The piece of code is a “lambda function” • JS (Node), Python, Java
53. def my_handler(event, context): message = 'Hello {} {}!'.format( event['first_name'], event['last_name']) return { 'message' : message }
54. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”) • The piece of code is a “lambda function” • JS (Node), Python, Java • A (Node) process in a container running the snippet behind the scenes
55. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”) • The piece of code is a “lambda function” • JS (Node), Python, Java • A (Node) process in a container running the snippet behind the scenes • Basically CGI
56. AWS Lambda 101 • Run code in response to events (S3, Dynamo etc) • You give AWS a snippet of code • AWS takes care of running it (“serverless”) • The piece of code is a “lambda function” • JS (Node), Python, Java • A (Node) process in a container running the snippet behind the scenes • Basically CGI • 300s max execution, up to 100 concurrently
57. AWS Lambda 101 • Access to AWS SDK
58. AWS Lambda 101 • Access to AWS SDK • Can be run on a schedule
59. AWS Lambda 101 • Perfect for scripting things in your AWS environment
60. AWS Lambda 101 • Perfect for scripting things in your AWS environment • e.g. DIY dynamic DNS system for instances not behind an ELB
61. AWS Lambda 101 • Perfect for scripting things in your AWS environment • e.g. DIY dynamic DNS system for instances not behind an ELB • Also great for small one-off tasks • E.g. resizing images uploaded to S3
62. AWS Lambda 101 • Perfect for scripting things in your AWS environment • e.g. DIY dynamic DNS system for instances not behind an ELB • Also great for small one-off tasks • E.g. resizing images uploaded to S3 • People also building APIs with it
63. Chaos Llama
64. Chaos Llama • Install: npm install –g llama-cli
65. Chaos Llama • Install: npm install –g llama-cli • Configure:
66. Chaos Llama • Install: npm install –g llama-cli • Configure: • Create a role for the lambda function
68. Chaos Llama • Install: npm install –g llama-cli • Configure: • Create a role for the lambda function • Deploy! llama deploy -r $lambda-role-arn
69. Chaos Llama • Install: npm install –g llama-cli • Configure: • Create a role for the lambda function • Deploy! llama deploy -r $lambda-role-arn • This is SAFE
71. Chaos Llama • Install: npm install –g llama-cli • Configure: • Create a role for the lambda function • Deploy! llama deploy -r $lambda-role-arn • This is SAFE • Llama will do nothing by default
72. // Llamafile.json: { "interval": "60", "enableForASGs": [ ], "disableForASGs": [ ] }
73. Going further
75. Going further • Increase network latency (temporarily)
76. Going further • Increase network latency (temporarily) • Increase memory usage
77. Going further • Increase network latency (temporarily) • Increase memory usage • Increase CPU load
78. Going further • Increase network latency (temporarily) • Increase memory usage • Increase CPU load • Can’t do this from a lambda function L
79. Llama Agent
80. Llama Agent • Simple RPC server (written in Go)
81. Llama Agent • Simple RPC server (written in Go) • Listens on a port; accepts (whitelisted) commands
82. Llama Agent • Simple RPC server (written in Go) • Listens on a port; accepts (whitelisted) commands • Commands correspond to shell scripts
83. sudo tc qdisc add dev eth0 root netem delay $1
84. Llama Agent • Simple RPC server (written in Go) • Listens on a port; accepts (whitelisted) commands • Commands correspond to shell scripts • Easy to install on a server
85. Llama Agent - Security • Small codebase (<100 LoC) • Memory-safe (Go) – no RCE
86. Llama Agent - Security • Small codebase (<100 LoC) • Memory-safe (Go) – no RCE • Whitelisted commands – no shell injections
87. Llama Agent - Security • Small codebase (<100 LoC) • Memory-safe (Go) – no RCE • Whitelisted commands – no shell injections • Restrict access to within a VPC only
88. Llama Agent - Security • Small codebase (<100 LoC) • Memory-safe (Go) – no RCE • Whitelisted commands – no shell injections • Restrict access to within a VPC only • The worst that can happen?
89. Llama Agent - Security • Small codebase (<100 LoC) • Memory-safe (Go) – no RCE • Whitelisted commands – no shell injections • Restrict access to within a VPC only • The worst that can happen? • Very experimental – not for production use yet
90. Summary • Chaos Engineering = more resilient systems
91. Summary • Chaos Engineering = more resilient systems • Chaos Llama is a modern take on Chaos Monkey
92. Summary • Chaos Engineering = more resilient systems • Chaos Llama is a modern take on Chaos Monkey • Easier to run thanks to AWS Lambda
93. Summary • Chaos Engineering = more resilient systems • Chaos Llama is a modern take on Chaos Monkey • Easier to run thanks to AWS Lambda
94. More • Toxyproxy – simulate faulty networks • Artillery.io – load-test HTTP & WebSockets
95. Questions
96. Questions
97. Extra: Money Llama • Untagged instances (story)
98. Extra: Money Llama • Untagged instances (story) • Detached EBS volumes (story)
99. Extra: Money Llama • Untagged instances (story) • Detached EBS volumes (story) • Cycle dev environments on a schedule