Training Engineering Teams with Chaos Engineering

Transcript

1. Training Engineering Teams with Chaos Engineering
2. YURY NIÑO ROA Site Reliability Engineer Chaos Engineering Advocate @yurynino https://www.yurynino.dev/
3. AGENDA Topics will be covered Motivations • Identifying needs • Training Engineers How to • Trainer Role • Instruction Principles GameDays • Chaos Engineering • Chaos GameDays https://www.yurynino.dev/
4. Identifying needs Training Engineers Motivations https://www.yurynino.dev/
5. Humans, are central to both the problem and the solution of challenges in engineering! https://www.yurynino.dev/
6. Identify Needs Needs Avoid Toil NaLSD Curiosity https://www.yurynino.dev/
7. Organizations Training SREs Training SREs https://www.yurynino.dev/
8. https://www.yurynino.dev/
9. • Be prepared. • State your objectives. • Be organized. • Use visuals. • Answer questions. • Be enthusiastic. Training Do’s • Provide feedback. • Be flexible. • Prepare for emergencies. • Encourage participation. • Establish rapport. • Be yourself. https://www.yurynino.dev/
10. • Starting late and wasting time. • Being poorly prepared and lacking knowledge. • Displaying distracting habits. • Ignoring participants and interrupting their questions. • Lacking enthusiasm. • Reading from a script. Mistakes to avoid https://www.yurynino.dev/
11. • Establish an informal atmosphere. • Encourage participants to take control. • Accept participants where they are. • Communicate openly and honestly. • Tap participants for the ideas. How to Ensure the Participation https://www.yurynino.dev/
12. • To be able to construct a mental representation. • To be able to assess risks and threats as relevant. • To be able to switch from a situation under control. • To be able to maintain a relevant level of confidence. • To be able to make a decision in a complex situation. In an emergency https://www.yurynino.dev/
13. • To be able to make an intelligent usage of procedures. • To be able to use available resources. • To be able to manage time and pressure. • To be able to cooperate with and crew members. • To be able to properly use and manage information. In an emergency https://www.yurynino.dev/
14. Human Factors
15. https://www.yurynino.dev/
16. Humans operate differently when they expect things to fail! Aaron Rinehart
17. Training Roles Instructions Principles How to https://www.yurynino.dev/
18. The world is an imperfect place. We can not control the environment! But we can control how to face the failures. https://www.yurynino.dev/
19. 4 Essential Capabilities 4 Sets of answers to construct resilience profile https://www.yurynino.dev/
20. Chaos Engineering Chaos GameDays GameDays https://www.yurynino.dev/
21. Everything fails, all the time! Werner Vogels https://www.yurynino.dev/
22. What is Chaos Engineering? It is the discipline of experimenting failures in production in order to reveal their weakness and to build confidence in their resilience capability. https://principlesofchaos.org/
23. What is Chaos Engineering? It is a scientific method that consists in specifying and evaluating resilience hypotheses 1) injecting faults in production 2) observing the impact 3) building resilience https://principlesofchaos.org/
24. 2008 Chaos Engineering began at Netflix 2010 Chaos Monkey & Simian Army were launched 2016 Gremlin born 2019 1 Book Chaos massification 2017 SRE Usenix Chaos IQ born ChaosConf 2018 1 Book Chaos Monkey for Spring Boot 2020 1 Book was published Chaos Engineering History
25. 1. Pick a Hypothesis: Recipe! 2. Choose the tools: Ingredients! 3. Launch an attack: Cook! 4. Notify the Org: Invite! 5. Run the Experiment: Enjoy! 6. Analyze the Results 7. Automate Chaos Principles
26. Hypothesize about Steady State Run Experiments Vary Real-World Events Automate Experiments Chaos: Principles
27. Chaos: Plan
28. https://github.com/yurynino/learning-chaos-springboot Chaos: Hypothesis
29. https://github.com/yurynino/learning-chaos-springboot Chaos: Running
30. Chaos: Running
31. Chaos: Running
32. Chaos: Running
33. Chaos: Running
34. Chaos: Running
35. Chaos: Automate
36. Chaos Toolkit Gremlin Chaos Engineering Tools
37. Before After During • Pick a hypothesis. • Pick a style. • Decide who. • Decide where. • Decide when. • Document. • Get approval! • Detect the situation. • Take a deep breath. • Communicate. • Visit dashboards. • Analyze data. • Propose solutions. • Apply and solve! • Write a postmortem. • What Happened • Impact • Duration • Resolution Time • Resolution • Timeline • Action Items Chaos Methodology
38. Chaos Engineering Science
39. The infrastructure required by a software system can be as complex as the software itself. We need a hands-on guide to exploring the world of Chaos! Netflix Twitter Chaos Engineering Motivations
40. From Chaos Engineering Book 2020 Chaos Maturity Model
41. Who are practicing?
42. Who are practicing
43. Disaster Piece Whenever they launch features or make changes, we test the fault tolerance of that new code! In January of 2018, they started a rigorous process of identifying failures that are likely to happen and that we must be able to tolerate, and then purposely causing them to happen in production. This isn’t Chaos Engineering as practiced and evangelized by Netflix. It’s the first step; we call it Disasterpiece Theater. Who are practicing
44. LinkedOut Taken from Chaos Engineering Book 2020 Taken from Chaos Engineering Book 2020 Who are practicing
45. Evolution CI/CD Tooling Culture Evangelism Team Taken from Chaos Engineering Book 2020 Who are practicing
46. Chaos GameDays GameDays are an interactive, real-world and learning exercises. They are designed to give players a chance to put their skills in a technology to test. GameDays were created by Jesse Robbins inspired by his experience & training as a firefighter.
47. GameDays Chaos Gamedays GameDays are interactive team-based learning exercises designed to give players a chance to put their skills to the test in a real-world, gamified, risk-free environment. A Chaos GameDay is a practice event, and although it can take a whole day, it usually requires only a few hours. The goal of a GameDay is to practice how you, your team, and your supporting systems deal with real-world turbulent conditions. Chaos References
48. First on Call Monitors, triages, and tries to mitigate failures caused by the Master of Disaster. Master of Disaster Decides the failure and declares start of incident and attack!!! Team Find and solve the exhibited issues, and write up postmortem. Chaos GameDays
49. Chaos References
50. Chaos References
51. https://chaosengineering.slack.com https://github.com/dastergon/ awesome-chaos-engineering https://www.infoq.com/chaos-engineering @yurynino How to begin?
52. Picasso Napkin Mark Manson .. The Subtle Art of Not Giving a F*ck
53. Thanks for coming!!! @yurynino https://www.yurynino.dev/