Runbookに何を書き、どのようにアラートを振り分けるか？

Transcript

1. RunbookʹԿΛॻ͖ɺ
   ͲͷΑ͏ʹΞϥʔτΛৼΓ෼͚Δ͔ʁ
   2023/09/29 SRE NEXT 2023
   Sohei Iwahori (GREE, Inc.)
   

   View Slide

2. Agenda
   » Introduction
   » Runbookͷ੔උ
   » ԿΛॻ͘΂͖͔ʁ
   » Runbookͷ࣮૷
   » Ξϥʔτ௥ՃΨΠυϥΠϯͷࡦఆ
   » Recap
   

   View Slide

3. Introduction
   

   View Slide

4. άϦʔʹ͓͚ΔΠϯϑϥ؀ڥͷมભ
   » ΦϯϓϨ͔ΒΫϥ΢υʢ2014ࠒʙʣ
   » VM͔Βίϯςφ΁ʢ2019ࠒʙʣ
   » ݱঢ়͸͜ΕΒͷ؀ڥ͕͢΂ͯଘࡏ
   » ΦϯϓϨ / VMϕʔεʢEC2 on AWSʣ / ίϯςφϕʔεʢEKS/GKEʣ
   

   View Slide

5. ήʔϜܥͷϫʔΫϩʔυͷಛ௃
   » αʔόαΠυ͸APIఏڙ͕ϝΠϯ
   » αʔϏεؒͷAPI࿈ܞ͸͋·Γͳ͍ʢ՝ۚܥͳͲڞ௨ػೳͱͷ΍ΓͱΓͷΈʣ
   » λΠτϧ͝ͱɺ৔߹ʹΑͬͯ͸ͦͷதͰ͞Βʹւ֎αʔϏε͝ͱͷ؀ڥΛ෼͚
   ͍ͯΔ
   » ݁Ռগͣͭ͠ҧ͏ɺࣅͨΑ͏ͳϫʔΫϩʔυͷ؀ڥ͕਺ेଘࡏ͢Δ͜ͱʹͳΔ
   » Ξϥʔτϧʔϧ͸ڞ௨Խ͍ͨ͠ͷͰԣஅͰઃఆͭͭ͠ɺҰ෦Λݸผઃఆ
   

   View Slide

6. DevͱOpsͷ૊৫ߏ଄
   » ࣄۀ෦͝ͱͷ։ൃνʔϜʹ਺ਓͷΠϯϑϥϝϯόʔ͕ͭ͘ܗ͕ࣜجຊ
   » ڞ௨ࢧԉ෦໳ͱͯ͠ͷΠϯϑϥ͕͍ͯɺͦͷதͰunitͱݺ͹ΕΔઐ໳ͷνʔ
   Ϝ͕͋Γɺڞ௨ج൫ͷఏڙʢ؂ࢹͳͲʣɺϛυϧ΢ΣΞʢRDBMSɺKVSͳ
   ͲʣͷࢧԉͳͲΛߦ͏
   » Πϯϑϥ಺ͷӡ༻νʔϜ͕ଘࡏ͠Ұ࣍ରԠΛ੥͚ෛ͏
   

   View Slide

7. DevͱOpsͷ૊৫ߏ଄
   

   View Slide

8. యܕతͳো֐ରԠϑϩʔ
   » PagerDuty/SlackͰো֐ͷൃੜΛ௨஌
   » ӡ༻νʔϜͰҰ࣍ରԠ࣮ࢪ
   » ো֐ରԠखॱॻʹج͍ͮͯఆܕతͳରԠΛ࣮ࢪ͢Δ
   » ղফ͠ͳ͍৔߹͸ϓϩμΫτʹ͍͍ͭͯΔΠϯϑϥΤϯδχΞɺ
   ·ͨ͸unitʢઐ໳νʔϜʣ΁ΤεΧϨ
   » EKS/GKE؀ڥʹ͓͍ͯ͸ӡ༻νʔϜΛܦ༝ͤͣ௚઀୲౰νʔϜ΁௨஌
   ͢ΔΑ͏ʹͳ͖͍ͬͯͯΔ
   

   View Slide

9. యܕతͳো֐ରԠϑϩʔʢΤεΧϨʣ
   

   View Slide

10. ʮো֐ରԠखॱॻʯͷ՝୊
    » ΤεΧϨͱͳͬͨ৔߹ʹରԠνʔϜ͕ࢀরग़དྷΔ৘ใ͕ͳ͍
    » ΤεΧϨઌ͸͋Δఔ౓஌ࣝΛ࣋ͬͨલఏͰ͸͋Δ͕ɺਓͷೖΕସΘΓ΋౰વ͋Δ
    » ݕࡧੑͷ໰୊
    » ConfluenceͰ1ϖʔδʹهࡌ͞ΕͨλΠτϧϕʔεͰͷݕࡧ
    » Ξϥʔτࣗମͷݟ௚͠ͷ೉͠͞
    » ରԠํ๏ʢHowʣʹͷΈϑΥʔΧε͍ͯ͠ΔͷͰͳͥ͜ͷΞϥʔτ͕
    ͋Δͷ͔(Why)͕Θ͔Βͳ͍
    

    View Slide

11. ো֐ରԠखॱͷݕࡧʢΠϝʔδʣ
    

    View Slide

12. ʮΞϥʔτΛ௥Ճ͍ͨ͠ʯ͔Βੜ͕ͪ͡ͳ՝୊
    » ᐆດͳཁ݅
    » ໰୊͕ى͖ͨͷͰΞϥʔτΛઃఆ͍ͨ͠
    » ௨஌ํ๏ɺظ଴͢ΔΞΫγϣϯ͓ΑͼλΠϛϯά͸ᐆດͳ··
    » ඇରশੑ
    » ԣஅͰϧʔϧΛ௥Ճ͢ΔࡍΞϥʔτΛઃఆ͢ΔਓΞϥʔτΛड͚Δਓ͕Ұக͠ͳ͍
    » ͋Δ೔ಥવݟ஌Β͵ΞϥʔτΛड͚Δ͜ͱʹͳΔ
    » ίϯςΩετͷ૕ࣦ
    » ͕࣌ؒܦա͠ɺΞϥʔτΛ௥Ճͨ͠จ຺͕ࣦΘΕͯ͠·͏
    » ʮͳʹ͔େࣄͳཧ༝͕͋ͬͨ͸ͣɾɾʯ
    

    View Slide

13. ͜ΕΒͷ՝୊Λղܾ͢ΔͨΊͷΞΫγϣϯ
    » ৽نʹʮΤεΧϨઌͷ୲౰ऀʯΛओͳλʔήοτͱͨ͠RunbookΛ੔උ
    » ΞϥʔτΛ௥Ճ͢ΔࡍͷϑϩʔɺΨΠυϥΠϯΛ࡞੒
    » ॱʹΈ͍͖ͯ·͢
    

    View Slide

14. Runbookͷ੔උ
    

    View Slide

15. ԿΛॻ͘΂͖͔ʁ
    

    View Slide

16. Runbookʹ͍ͭͯͷ༷ʑͳҙݟ(1/3)
    practiced on-call engineer armed with a playbook works much better.
    — Site Reliability Engineering /Chapter 1 - Introduction
    playbook(runbook)ͷਪ঑
    

    View Slide

17. Runbookʹ͍ͭͯͷ༷ʑͳҙݟ(2/3)
    Just like new code, new alerts should be thoroughly and thoughtfully reviewed.
    Each alert should have a corresponding playbook entry.
    — SRE Workbook / Chapter 8 - On-Call
    ΞϥʔτʹରԠͨ͠playbook(runbook)ͷΤϯτϦ͕͋Δ΂͖
    

    View Slide

18. Runbookʹ͍ͭͯͷ༷ʑͳҙݟ(3/3)
    The assertion that time spent creating runbooks is largely wasted may seem a bit
    harsh at first.
    — Observability Engineering / 8. Analyzing Events to Achieve
    Observability
    runbookʹ͔͚Δ࣌ؒ͸ແବ͔΋͠Εͳ͍
    

    View Slide

19. Runbook͕༗ޮʹػೳ͢Δ৚݅Λߟ͑ͯΈΔ
    » ஌ݟͷੵΈ্͕͛Մೳͳ؀ڥ
    » ࣅͨΑ͏ͳϫʔΫϩʔυ͕ෳ਺͋Δ
    » ϕʔεٕज़ͷมԽ͕ΏΔ΍͔
    » ৗʹ࢐৽ͳʢ࠶ݱੑͷͳ͍ʣΞϥʔτ͕ൃੜ͢ΔΑ͏ͳ؀ڥʹ͸ෆ޲͖
    » ͓ͦΒ͘ΦϒβʔόϏϦςΟΤϯδχΞతͳΞϓϩʔν͕ඞཁ
    » ϗϫΠτϘοΫεϕʔεͷΞϥʔτΛར༻/ซ༻͍ͯ͠Δ
    

    View Slide

20. RunbookʹԿΛॻ͘΂͖͔ʁʹ͍ͭͯͷҙݟ
    » ୹ظతͳղܾΛࢤ޲͢Δ͔ɺ௕ظతͳରԠ΋ؚΊͨώϯτ͔
    » εςοϓόΠεςοϓͷखॱࢤ޲͔ɺશମతͳ஌ࣝͷڞ༗ࢤ޲͔
    » ϒϥοΫϘοΫεϕʔεͷΞϥʔτʹରͯ͠͸ͦ΋ͦ΋खॱͷఏ͕ࣔ೉͍͠
    ͷͰ͸ͳ͍͔
    

    View Slide

21. άϦʔʹ͓͚ΔRunbookࢤ޲
    » ৆ຯظݶͷ୹͍୹ظతҊղܾࡦΑΓ௕࣋ͪ͢Δ৘ใ
    » എܠͷڞ༗Λॏࢹ
    » ΤϯδχΞʹΑΔ൑அɺΞΫγϣϯʹͭͳ͕Δ৘ใ
    » ରԠʹ͸ΤϯδχΞͷ஌ੑ͕ඞཁͳ͸ͣ
    » ղܾࡦ͕ఆܕతͳίϚϯυͰ͋Ε͹ࣗಈԽͰΑ͍
    » ղܾʹͭͳ͕ΔίϯςΩετ / ώϯτΛఏࣔ͢΂͖
    

    View Slide

22. Runbookͷ࣮૷
    

    View Slide

23. RunbookςϯϓϨʔτ
    

    View Slide

24. Runbookαϯϓϧ
    

    View Slide

25. Runbookαϯϓϧ
    

    View Slide

26. ࣮૷
    » gitϨϙδτϦӡ༻
    » Ξϥʔτ௨஌γεςϜ΁ͷ૊ΈࠐΈ
    » ඞཁͳλΠϛϯάͰίϯςΩετΛิ׬͢Δ
    » ϢϏΩλευΩϡϝϯτతΞϓϩʔνΛߟ͑Δ1
    » ؆қͳTemplate͔Βͷ࡞੒
    1 Low Context DevOps: Improving SRE Team Culture through Defaults, Documentation, and Discipline Tom Limoncelli / SREcon20
    Americas.
    

    View Slide

27. Runbookͷ൓өϑϩʔ
    

    View Slide

28. Ξϥʔτ௨஌γεςϜ΁ͷ૊ΈࠐΈ
    

    View Slide

29. Runbook͕ଘࡏ͠ͳ͍৔߹͸࡞੒Λଅ͢
    

    View Slide

30. ޮՌͱ՝୊
    » ʮ׬શͳະ஌ͷΞϥʔτʯΛड͚ΔػձΛݮΒ͢͜ͱ͕Ͱ͖ͨ
    » ৺ཧత҆શੑ
    » ΦϯϘʔσΟϯάͰ࿫͍͑ͯͳ͍෦෼ͷαϙʔτ
    » Ҿ͖ଓ͖՝୊ͱײ͡Δ΋ͷ
    » υΩϡϝϯτࣗମͷޮՌଌఆ
    » ఆظతͳϝϯςφϯεͷΩοΫ
    » ྫ͑͹ҰఆظؒͰauthorʹ௨஌͕͍͘࢓૊ΈͳͲ
    

    View Slide

31. Ξϥʔτ௥Ճ
    ΨΠυϥΠϯͷࡦఆ
    

    View Slide

32. νϟϯωϧબ୒ͱ
    ԣஅΞϥʔτͷͨΊͷΨΠυϥΠϯ
    

    View Slide

33. νϟϯωϧબ୒ΨΠυϥΠϯ
    

    View Slide

34. ԣஅΞϥʔτ௥ՃͷΨΠυϥΠϯ
    

    View Slide

35. ௥Ճϑϩʔ΁ͷ૊ΈࠐΈ
    » ௥Ճϑϩʔʹ૊ΈࠐΉ
    » ௥Ճ͍ͨ͠ɺͱͳͬͨ৔߹ʹࣗવͱҎԼͷ͜ͱΛߟ͑
    Δඞཁ͕͋Δ
    » ૝ఆνϟϯωϧ=͍ͭରԠ͢΂͖΋ͷ͔
    » είʔϓ=ͲͷൣғͰద༻͢΂͖͔
    

    View Slide

36. ΨΠυϥΠϯࡦఆʹΑΔޮՌ
    » ΞϥʔτΛઃఆ͢Δஈ֊ͰΞΫγϣϯΛݻΊΒΕΔ
    » ͲͷλΠϛϯάͰ
    » Ͳͷ௨஌νϟϯωϧΛ࢖ͬͯ
    » ͲͷΑ͏ʹߦ͏͔
    » ΞΫγϣϯΛߟ͑Δ͜ͱͰΞϥʔτࣗମͷඞཁੑ΋࠶ݕ౼Ͱ͖Δ
    » ΞϥʔτർΕͷܰݮ
    

    View Slide

37. Recap
    

    View Slide

38. Recap
    » Runbook͕͏·͘ϫʔΫ͢ΔͨΊͷ৚݅
    » ੵΈ্͕͛Մೳ / ϗϫΠτϘοΫεϕʔεͷ௨஌Λར༻
    » എܠɺίϯςΩετΛ௕ظతʹ఻͑Δ͜ͱ͸ॏཁ
    » Ξϥʔτͷઃఆ࣌ͷΨΠυ͸ରԠΞΫγϣϯΛ۩ମతʹߟ͑ΔͨΊʹ༗ޮ
    » ࣮૷༗ແɺܗଶ͸૊৫ͷ՝୊ʹ߹Θͤͯબ୒͢Δͱྑ͍
    

    View Slide

39. Thank you for listening
    

    View Slide

40. who?
    » Sohei Iwahori (@egmc)
    » GREE, Inc.
    » Πϯϑϥ / Monitoring Unit Leader
    » ήʔϜͷΠϯϑϥͱϞχλϦϯά
    

    View Slide

41. Appendix
    » Low Context DevOps: Improving SRE Team Culture through Defaults,
    Documentation, and Discipline
    » https://www.usenix.org/conference/srecon20americas/presentation/
    limoncelli
    » GitLab On-call Run Books
    » https://gitlab.com/gitlab-com/runbooks
    » Dashboards and Runbooks: Scrapbooking for Engineers
    » https://www.usenix.org/conference/srecon22apac/presentation/douch
    

    View Slide

42. Appendix
    » ΦϯίʔϧΞϥʔτΞϯνύλʔϯ
    » https://dasalog.hatenablog.jp/entry/2022/05/23/141749
    

    View Slide