Observing and Understanding Failures - SRE Apprentices

Transcript

1. Observing & Understanding Failures Training SRE Apprentices Tammy Bryant Butow, SRE @ Gremlin @tambryantbutow
2. Is it difficult to develop skills to observe and understand failures? Why is training from someone more experienced helpful? @tambryantbutow
3. @tambryantbutow Luke before he met Yoda Could barely use force
4. Luke after he met Yoda Lifts ship out of swamp after doing a handstand @tambryantbutow
5. SRE Apprentice @tambryantbutow Padawan We created an SRE Apprentice Program to hire and train new SREs. Apprentices come from different backgrounds, for example we hired a Math Teacher who had completed a coding bootcamp (Hackbright Academy). We matched SRE Apprentices with SRE Teachers.
6. SRE Apprentice @tambryantbutow SRE Teacher Padawan Jedi SRE Apprentices receive one-on-one instruction in the ways of the SRE.
7. @tambryantbutow When an Apprentice’s training is completed, they must pass the SRE interview-loop to become an SRE. They then continue to develop their skills and one day they will find a Padawan to train. SRE Teacher Padawan Jedi SRE Apprentice
8. Let’s check our assumptions on SRE Apprentices @tambryantbutow Rona Krishelle Thomissa John
9. How do we match SRE Apprentices (Padawans) to SRE Teachers (Jedi)? @tambryantbutow
10. SRE Apprentice @tambryantbutow Padawan - Rona Chong The SRE apprenticeship was critical for my career - it was my foot into the door of the tech industry, when it can be hard to break in as a newcomer without the usual credentials. But getting your foot in the door is just the first step. To help set me up for success, Tammy was several things for me: a mentor, a source of emotional support, and an advocate. Find people who are truly dedicated to being there for others! I'm so glad that Tammy and her folks were there for me, checking in, thinking outside the box and pushing for growth and change in our communities - Rona Chong, Padawan // SRE Apprentice.
11. SRE Apprentice @tambryantbutow SRE Teacher Padawan - Rona Chong Jedi - Tammy Bryant Butow Can feel disturbances in the force and use skills effectively to achieve desired results. Has a desire to learn everything she can about the force.
12. SRE Teacher @tambryantbutow What makes someone suitable? • Set your Apprentice up for success • Be a mentor • Be a source of emotional support • Be an advocate • Check-in • Think outside the box • Push for growth and change in our communities • 2+ years SRE experience
13. Psychological Safety @tambryantbutow What is the S.A.F.E.T.Y model and how can it help me teach my apprentice? Self Assessment: https://academy-bbl.com/safety-assessment/
14. SRE Apprentice @tambryantbutow Padawan An approach to learning to observe and understand failures in Production Learn by training Learn by shadowing Learn by practice Learn by community * Mandalorian-style apprentice program featuring Grogu aka Baby Yoda and a Jedi trainer
15. SRE Apprentice Learn by training: start with demo apps & lab environments before production @tambryantbutow
16. burningion/ecommerce-observability @tambryantbutow
17. localhost:3000/cart?variant_id=5 ads service code is throwing errors from a specific erb file @tambryantbutow
18. discounts-service/discounts.py discounts service code is causing performance issues @tambryantbutow
19. gremlin/microservices-demo @tambryantbutow
20. Architecture 12 Services - What matters most to our customers? @tammyxbryant
21. What can we remove from the critical path? @tammyxbryant Architecture
22. @tambryantbutow OH: “Getting out of the critical path is a good thing” gtk
23. Service Not Found Architecture @tammyxbryant Does blackholing a non-critical path service like the Recommendation Service cause unexpected failures for critical services like the Product Catalogue or Frontend?
24. Blackhole → Ads @tambryantbutow
25. @tambryantbutow
26. @tambryantbutow
27. @tambryantbutow Does blackholing a non-critical path service like the Ad Service result in graceful degradation of the customer experience?
28. @tambryantbutow
29. Graceful Degradation @tambryantbutow Yes, our experiment was successful and our results were what we expected them to be.
30. SRE Apprentice Learn by shadowing: on-call @tambryantbutow
31. twitter.com/tambryantbutow/status/1387778372688830465 @tambryantbutow
32. twitter.com/tambryantbutow/status/1387778372688830465 @tambryantbutow
33. @tambryantbutow SRE Apprentice on-call shadow role On-Call Rotation Example
34. @secnerdette @tambryantbutow
35. Partner with your local community to teach the required skills engineers need @tambryantbutow holbertonschool.com
36. SRE Apprentice Learn by practice: code + code reviews @tambryantbutow
37. Partner with your local community to teach the required skills engineers need @tambryantbutow
38. @tambryantbutow ✉
39. Craft a daily email using python to send via cron job that includes key metrics; disk capacity, availability, latency etc. @tambryantbutow
40. Build a web page that uses the PagerDuty API to display the most common alerts ordered by frequency - this enables us to use the Pareto Principle to improve system reliability. @tambryantbutow
41. SRE Apprentice Learn by community: peer mentoring, lunch and learns, study hall, AMAs & slack @tambryantbutow
42. @tambryantbutow 7000+ engineers # learning # donut-intros # questions gremlin.com/slack
43. SRE Apprentices share tips for mentors @tambryantbutow
44. @tambryantbutow Tips for SRE Teachers: 1. Meet your apprentice where they are and help them get where they need to be. 2. Don't limit yourself as a mentor: Your apprentice's role isn't limited to code, so why should your mentorship? 3. Don't forget to establish open lanes of communication.
45. @tambryantbutow funretrospectives.com/draw-your-feelings/
46. SRE Apprentices share tips for future padawans @tambryantbutow
47. @tambryantbutow . Tips for Apprentices: 1. One of the secrets to being effective is being willing to ask questions. 2. Don't just accept a task. 3. Unrealistic expectations happen all the time in software. 4. Invest in your relationships with your colleagues
48. Thank you @tambryantbutow Get some chaos community stickers based on the path you’ll be taking: Padawan: gremlin.com/talk/padawan Jedi: gremlin.com/talk/jedi