chaos_carnival.pdf

Transcript

1. Chaos Session
2. Documenting the Chaos
3. Yury Niño Roa Cloud Infrastructure Engineer Site Reliability Engineer Chaos Engineering Advocate @yurynino www.yurynino.dev
4. Agenda * A Chaos’s Story * Documentation is Important * Documentation Framework ** Docs for General Chaos ** Docs for Preparing Chaos ** Docs for Running Chaos ** Docs for After the Chaos
5. A Chaos Story in the middle of Chaos
6. Chaos in the middle of Chaos Before the Chaos Friday 13:50 - Planned A co-host of Chaos Experiment announces a routine of chaos in the #chaos-eng Slack channel. During the Chaos Friday 14:00 - Planned One of the co-hosts runs the prepared commands to incite the failure in PROD environment. Note taker record the time. During the Chaos Friday 14:15 - Planned All seems to be well, the team collects the evidence of the failure, confirms the hypothesis and prepares the recovery. Real Chaos Friday 14:30 - NO Planned PROD environment should return to a steady state but something is wrong.. After the Chaos Friday 16:05 - Planned An automated recovery procedure put online the region with chaos. After 90 min Team finds a document with commands to restart the new service. They cross their fingers and fortunately it works! Real Chaos Friday 14:50 - NO Planned In spite of having experience, the failure is on a new component and there is not documentation available! www.yurynino.dev
7. Recipe for the Chaos in the middle of Controlled Chaos www.yurynino.dev
8. Luckily, all the characters and episodes in this story are fictional. Things that went well: team work, use of communication channels and chaos engineering isolation. Things that did not go well: lack of knowledge about the automation recovery script, dev team unavailable and lack of documentation! Action item priority Meeting between Dev and SRE team Documentate the new service PostMortem Time www.yurynino.dev
9. Documentation is Important!
10. Because the Chaos Engineer's work is never done. • Organizations can not depend on the knowledge of individuals passed verbally to new members. • Documentation helps developers communicate with each other. • Documents help future developers understand and maintain the code. • Good documentation help you learn from your mistakes. Documentation is important because … If the concepts are not documented, they will need to be relearned painfully through trial and error like the previous story!
11. Role of Technical Writers
12. • Writers should partner with CEs to provide operational documentation for running services and product documentation. • Writers should provide consulting to assess, assist, and address documentation and information management needs. • Writers should evaluate and improve documentation tools to provide the best solutions for Chaos Engineering. Technical Writers
13. SRE teams can prevent this process decay by creating high-quality documentation that lays the foundation for such teams to scale up and take a principled approach for managing new and unfamiliar services. www.yurynino.dev
14. Documentation Framework
15. Chaos Designs Playbooks Incident Management Chaos Documentation Framework General Team Charter Production Readiness Technical Designs Before Chaos Policies Service Agreements On Call Policies During After Postmortems Reliability Reports
16. General Documents
17. A charter establishes the identity, primary goals, and roles in the team. Team Charter . Team Charter . How team operates Vision statement Short description of top services Key principles and values Links to the team site and docs www.yurynino.dev
18. TDDs are similar to proposals, that describe how a specific solution will function. www.yurynino.dev Technical Design Document . Technical Design . System Overview System Architecture Infrastructure Services Documentation Standards Naming conventions
19. TDDs are similar to proposals, that describe how a specific solution will function. www.yurynino.dev Technical Design Document . Technical Design . Programming Standards Development tools Requirements Traceability Matrix Document Control Document Signoff Document Change Record
20. A PRR examines a program to determine if the design is ready for production, www.yurynino.dev Production Readiness Review. Architecture and Dependencies Capacity Planning Failure Modes Processes and Automation External Dependencies Production Readiness Review
21. Documents for Before the Chaos
22. Policies are statements of intent implemented as procedures or protocols. www.yurynino.dev Chaos Policies . Chaos Policies . Overview Policy Goals Steady State SLOs & SLIs Key Policies Outage Policy Escalation Policy Related Documentation
23. Service Agreements. Scope User Target Internal Targets Supporting Documentation www.yurynino.dev A Service Agreement is a contract that sets the standard Terms and Conditions for Google Ads account budgets. Production Readiness Review
24. TDDs are similar to proposals, that describe how a specific solution will function. www.yurynino.dev OnCall Policies . OnCall Policies . Overview & Readiness Training & Scheduling Shift Details Pager Load Compensation Tools & Processes Communications Standards
25. Documents for During the Chaos
26. A Chaos Eng Design is one of the most important asset in the framework www.yurynino.dev Chaos Designs . Chaos Eng Designs . Application Name Hypothesis Environment Duration & Load Results Observability Actions
27. Playbooks. Overview Alert Severity Verification Troubleshooting Solution Escalation Related Links www.yurynino.dev A playbook lets oncall engineers respond to alerts generated by service monitoring. Playbooks
28. Documents for After the Chaos
29. Postmortems. Things that went well Things that didn’t go well What to improve for next time Lessons learned Action items Action item priority www.yurynino.dev A postmortem is an analysis conducted after a system failure. Postmortems
30. TDDs are similar to proposals, that describe how a specific solution will function. www.yurynino.dev Reliability Reports . Reliability Reports . Indicator name Collection method Assessment/formula/scale criteria Targets and performance thresholds Source of data Data frequency Data entry Expiry/revision date
31. Thank you!