備えあれば患いなし：効率的なインシデント対応を目指すSREの取り組み

Transcript

1. උ͑͋Ε͹ױ͍ͳ͠
   
   
   ~ޮ཰తͳΠϯγσϯτରԠΛ໨ࢦ͢SREͷऔΓ૊Έd
   SRE NEXT 2023
   
   
   2023-09-29
   גࣜձࣾϕΨίʔϙϨʔγϣϯ খݪҰਅ @kazumax55
   

   View Slide

2. ࣗݾ঺հ
   খݪҰਅ(Kazuma Kohara)
   
   
   @kazumax55
   γεςϜ౷ׅ෦ SRE෦ ෦௕
   
   
   (SRE෦ = SRE group + Corporate IT group)
   झຯ
   
   
   🏕Ωϟϯϓ 🎣௼Γ 🧖α΢φ
   ԶͨͪͷSREͱNew Relic
   
   
   -ॻ੶ฤ-
   NRUG SREࢧ෦ ӡӦ
   
   
   https://nrug-sre.connpass.com/
   

   View Slide

3. LOWYAʢϩ΢Ϡʣͱ͸ʁ
   
   
   2004೥7݄ʹ෱ԬͰ૑ۀͨ͠Ո۩ɾΠϯςϦΞͷD2Cϒϥϯυ
   
   
   ঎඼։ൃ͔Βൢച·ͰΛҰؾ௨؏Ͱखֻ͚Δ͜ͱͰɺ
   
   
   τϨϯυੑͷߴ͍঎඼Λ͓ٻΊ΍͍͢Ձ֨Ͱ͓٬༷ʹఏڙ͍ͯ͠·͢ɻ
   
   
   2023೥ ෱Ԭʹॳͷ௚ӦళΛOPENʂ2߸ళ͸େࡕͳΜ͹ύʔΫεʹग़ళʂ
   

   View Slide

4. LOWYAͷಛ௃ɿD2Cத৺ͷϏδωεϞσϧͰίετϝϦοτΛސ٬ؐݩ
   LOWYA=ϩʔϓϥΠεͳ͓ళ
   
   
   ௕͍όϦϡʔνΣʔϯΛҰؾ௨؏Ͱ୲͍ɺτϨϯυੑͷߴ͍঎඼Λ͓ٻΊ΍͍͢Ձ֨ͰͷఏڙΛ࣮ݱ
   

   View Slide

5. ࠓ೔͓࿩͢Δࣄ
   ޮ཰తʹΠϯγσϯτରԠΛߦ͏ͨΊͷ
   
   
   औΓ૊Έʹ͍ͭͯ
   

   View Slide

6. LOWYAࣄۀͷΠϯγσϯτରԠϑϩʔ
   උ͑͋Ε͹ױ͍ͳ͠ʂ
   
   
   ࠓ೔͸ϙΠϯτͱͳΔ੺ࣈͷ෦෼ʹ͍ͭͯ
   
   
   औΓ૊Έͷ಺༰Λ͝঺հ͠·͢
   Πϯγσϯτ͕ऩଋͨ͠Βʁ
   
   
   ඇ೉ͷͳ͍ৼΓฦΓͱͯ͠PostmortemΛॻ͘
   Πϯγσϯτ͕ൃੜͨ͠Βʁ
   ·ͣSeverity LevelsΛ൑ఆ͢Δ
   
   
   SEVϨϕϧผEscalation FlowʹԊͬͯ৘ใΛΤεΧϨʔγϣϯ͢Δ
   
   
   ΠϯγσϯτରԠΛ࣮ࢪ͢Δ
   ͞Βʹɺ๨Εͨࠒʹൃੜ͢ΔΠϯγσϯτʹඋ͑ͯ
   
   
   ఆظతʹIncident Fire DrillΛ࣮ࢪ͠ɺ܇࿅ͱطଘϑϩʔͷ఺ݕΛߦ͏
   

   View Slide

7. Severity Levels
   

   View Slide

8. Severity Levels
   Severity Levelsͱ͸ʁ
   
   
   Πϯγσϯτͷॏେ౓/ਂࠁ౓Λද͢Ϩϕϧ
   
   
   Өڹͷେ͖͞ΛϨϕϧผʹݴޠԽͨ͠΋ͷ
   Sevelity LevelsΛఆٛ͢Δ໨త
   
   
   ॏେ౓ͷೝࣝΛἧ͑ɺਝ଎͔ͭద੾ͳରԠΛߦ͏
   
   
   ॏେ౓Λૉૣ͘൑அ͠ɺαʔϏε෮چʹΑΓଟ͘ͷ࣌ؒΛׂ͘
   

   View Slide

9. Severity Levels : ྫ
   Severity Description Typical Response
   SEV-1
   n෼Ҏ্ͷαʔϏεఀࢭ͔ͭ෮چͷ໨ॲཱ͕ͨͳ͍ঢ়گ
   
   
   αΠόʔ߈ܸΛड͚͔ͭಥഁ͞ΕΔ
   
   
   ػີ৘ใͷ࿙Ӯ
   SEV-2ʹՃ͑ɺ
   
   
   ձࣾͱͯ͠αʔϏεঢ়گͷ֎෦։ࣔΛݕ౼͢Δ
   SEV-2
   αʔϏεར༻͕Ͱ͖ͳ͍ϨϕϧͷύϑΥʔϚϯε௿Լ
   
   
   ػೳ͕ಈ࡞͠ͳ͍
   
   
   SLOͷܧଓతͳ௿Լ
   ෮چʹඞཁͳମ੍Λ੔උ͠ɺ࠷༏ઌͰରԠ
   
   
   αʔϏεར༻ऀʹ޲͚ͨΞφ΢ϯεΛݕ౼͢Δ
   ͜ΕΑΓ্ͷϨϕϧ͸ॏେΠϯγσϯτͯ͠ѻ͏
   SEV-3
   ରԠ͕஗ΕΔͱக໋తʹͳΔΠϯγσϯτ
   
   
   ϢʔβʔʹӨڹͷ͋ΔΤϥʔɺδϣϒͷࣦഊ
   
   
   SLOͷҰ࣌తͳ௿Լ
   ௨ৗλεΫΑΓ༏ઌͯ͠ௐࠪɾରԠ
   SEV-4
   ϢʔβʔʹӨڹͷ͋ΔܰඍͳΠϯγσϯτ
   
   
   ॲཧ஗Ԇ౳
   ௨ৗλεΫΑΓ༏ઌ౓ߴΊͰௐࠪ
   SEV-5 Ϣʔβʔʹ௚઀Өڹ͠ͳ͍ܰඍͳΠϯγσϯτ IssueΛཱͯͯରԠ
   ͍͟ͱ͍͏ͱ͖ରԠʹ໎Θͳ͍Α͏ɺͰ͖Δ͚ͩγϯϓϧʹఆٛ͢Δ
   

   View Slide

10. Severity Levels : ఆٛͷίπ
    ୭Ͱ΋༰қʹ൑ఆͰ͖ΔΑ͏ɺᐆດͳදݱΛආ͚ͯγϯϓϧʹ͢Δ
    
    
    ஗ԆͳΒSEV-4, ΤϥʔͳΒSEV-3 ౳
    
    
    ᐆດͳදݱ͸ආ͚Δ
    
    
    ॏཁͳػೳͷ… ͱ͔ɺ௕࣌ؒͷఀࢭ… Έ͍ͨͳදݱ
    
    
    ΍ΔͳΒ Կ෼Ҏ্ ͱ͔ɺॏେͳػೳ = AͱBͱC ͷΑ͏ʹܾΊΔ
    
    
    ໎ͬͨͱ͖͸ɺ্Ґͷॏେ౓ͱͯ͠ѻ͏
    
    
    ૣ͘Ϩϕϧ൑ఆͰ͖ΔΑ͏ʹ͢ΔͨΊͷϧʔϧ
    

    View Slide

11. Severity Levels : ࢀߟυΩϡϝϯτ
    PagerDutyɿIncident Respons > Before an Incident > Severity Levels
    
    
    https://response.pagerduty.com/before/severity_levels/
    
    
    AtlassianɿΠϯγσϯτͷॏେ౓Ϩϕϧͷ೺Ѳ
    
    
    https://www.atlassian.com/ja/incident-management/kpis/severity-levels
    ͱͯ΋ࢀߟʹͳΓ·͕͢ɺᐆດͳදݱ΋ଟ͘൑அʹ໎͏ࣄ͕͋ͬͨҝ
    
    
    ݟ௚͍ͯͬͨ݁͠ՌɺઌఔͷΑ͏ͳγϯϓϧͳϧʔϧʹམͪண͍ͨܦҢ͕͋Γ·͢
    
    
    ༨ஊɿSlackελϯϓΛ࡞͓ͬͯ͘ͱศར SEV-1, SEV-2, SEV-3, SEV-4, SEV-5
    

    View Slide

12. Escalation Flow
    

    View Slide

13. Escalation Flow
    ΤεΧϨʔγϣϯϑϩʔͱ͸ʁ
    
    
    ༗ࣄͷࡍͷ৘ใڞ༗ઌΛSEVϨϕϧผʹ༧Ίఆٛͨ͠΋ͷ
    ΤεΧϨʔγϣϯϑϩʔΛఆٛ͢Δ໨త
    
    
    ༗ࣄͷࡍʹ໎ΘͣεϜʔζͳ৘ใڞ༗Λߦ͏ͨΊ
    
    
    SEVϨϕϧผʹʮ୭ʹใࠂ͢Δ͔ʁʯʮνʔϜͰͲͷΑ͏ʹ໾ׂ෼୲ͯ͠ಈ͔͘?ʯΛఆ
    Ί͓ͯ͘
    ؔ܎ऀ୭͚ͩͬʁ
    ࿈བྷઌͲ͚ͩͬ͜ʁ
    

    View Slide

14. Escalation Flow : ྫ
    SEVϨϕϧຖͷΤεΧϨʔγϣϯઌͷఆٛ
    
    
    Πϯγσϯτใࠂ༻Slack channelΛ࡞Γؔ܎ऀΛট଴͠ɺॏཁͳ৘ใ͸ͦ͜ʹpinཹΊ͓ͯ͘͠
    
    
    PinཹΊਪ঑ɿΠϯγσϯτରԠ࣌ʹࢀর͢Δ֤छυΩϡϝϯτ, VIDEOձٞURL౳
    
    
    SEVϨϕϧผͷΤεΧϨʔγϣϯઌΛఆ͓ٛͯ͘͠
    
    
    όΠωʔϜͰ͸ͳ͘ɺ໾৬Ͱهࡌ͢ΔʢϝϯςφϯείετΛݮΒͨ͢Ί
    
    
    ۀ຿࣌ؒ֎ͷۓٸ࿈བྷઌϦετ
    
    
    Slackʹ൓Ԡ͕ͳ͍࣌ɺۓٸ࿈བྷઌѼʹ࿈བྷ͢Δͷ͔౳΋༧ΊܾΊͯ߹ҙΛऔΔ
    
    
    ༨ஊɿखಈPagerൃՐखஈ
    
    
    ૝ఆ֎ͷΠϯγσϯτൃੜ࣌ʹɺΦϯίʔϧ౰൪ΛSlack͔Βݺͼग़ͤΔΑ͏ͳ΋ͷ
    
    
    SlackͷWork
    fl
    owͰ࡞͓ͬͯ͘ͱศར
    

    View Slide

15. Escalation Flow : ໾ׂ෼୲
    ͞·͟·ͳ໾ׂ(PagerDuty͔ΒҾ༻)
    

    View Slide

16. Escalation Flow : ໾ׂ෼୲ͷࢀߟυΩϡϝϯτ
    PagerDutyɿIncident Respons > Before an Incident > Different Roles
    
    
    https://response.pagerduty.com/before/different_roles/
    
    
    PagerDutyɿIncident Respons > Training > Incident Commander
    
    
    https://response.pagerduty.com/training/incident_commander/
    
    
    New RelicɿNew RelicྲྀɺΦϯίʔϧͱΠϯγσϯτରԠͷ੒ޭ΁ͷಓ
    
    
    https://newrelic.com/jp/blog/how-to-relic/on-call-and-incident-response-new-relic-best-practices-2
    αΠτͷن໛ʹΑͬͯ͸ɺ໾ׂ෼୲ʹ͍ͭͯ΋ݚڀͯ͠ӡ༻ʹऔΓೖΕΒΕΔΑ͏τϨʔχϯά
    ͓ͯ͘͠ͱྑ͍Ͱ͠ΐ͏
    

    View Slide

17. Postmortem
    

    View Slide

18. Postmortem
    ϙετϞʔςϜͱ͸ʁ
    
    
    ΠϯγσϯτͷࣄޙରԠͱ࣮ͯ͠ࢪ͢Δඇ೉ͷͳ͍ৼΓฦΓ
    
    
    ϙετϞʔςϜͷ໨త
    
    
    ࣦഊ͔Βֶͼɺಉ͡աͪΛ܁Γฦ͞ͳ͍
    
    
    ࠜຊݪҼͷ֬ೝͱޮՌతͳ࠶ൃ๷ࢭࡦΛݕ౼͢Δ
    

    View Slide

19. Postmortem : ॏཁͳϙΠϯτ
    ඇ೉͠ͳ͍ɾ੹೚௥ٴͷ৔ʹ͠ͳ͍
    
    
    ϛεͨ͠ਓΛɺԜ·ͤͯ΋Կ΋͍͍͜ͱ͸ͳ͍
    
    
    લ޲͖ʹվળʹ͍ͭͯ࿩͠߹͏৔ʹ͢Δ
    
    
    ݪҼಛఆ΍࠶ൃ๷ࢭ͸࢓૊ΈͰߟ͑Δ
    
    
    ࠶ݱੑͷ͋ΔखॱΛಋ͖ग़͢ҝɺݸਓͷ஫ҙ΍ೳྗʹཔΒͳ͍
    
    
    ۩ମతͳνΣοΫ߲໨Λߟ͑ͳ͍ͱɺԿਓͰ֬ೝͯ͠΋ҙຯ͕ͳ͍
    
    
    NGϫʔυɿ͔ͬ͠Γɺ஫ҙਂ͘
    
    
    ಉ͡աͪΛ๷͙ʹ͸ɺԿ͕Ͳ͏͍͏ঢ়ଶͰ͋Δ͜ͱΛ֬ೝ͢΂͖͔
    
    
    ͞ΒʹͦΕΛ࢓૊Έͱཱͯ֬͠Ͱ͖ͳ͍͔ߟ͑Δ͜ͱ͕ॏཁ
    

    View Slide

20. Postmortem : ޻෉͍ͯ͠Δ఺
    ೋ౓ݟ͢Δ
    
    
    ि࣍։࠵͢ΔΤϯδχΞఆྫͰશମʹڞ༗
    
    
    ͞ΒʹSRE+ϓϩμΫτ։ൃνʔϜؒͰ݄࣍։࠵͢ΔSLO ReviewͰ࠶֬ೝ
    
    
    ࣌ؒΛ͓͍ͯ࠶֬ೝ͢Δͱɺ৽ͨͳٙ໰΍վળ఺͕ු͔ΜͰ͘Δࣄ͕͋Δ
    
    
    ςϯϓϨʔτʹΠϯγσϯτରԠϑϩʔͷৼΓฦΓ߲໨Λ௥Ճ
    
    
    SEV൑ఆ͍͖ͤͣͳΓमਖ਼Λ࢝Ίͯ͠·͏νʔϜʹɺϑϩʔΛ࠶ೝࣝͯ͠΋Β͏
    
    
    1. ॳಈͰSEV൑ఆͰ͖·͔ͨ͠ʁ
    
    
    2. ࣄલʹఆΊͨϑϩʔʹԊͬͯΤεΧϨʔγϣϯ͕ग़དྷ·͔ͨ͠ʁ
    
    
    ࠓޙ΍Γ͍ͨ͜ͱɿϙετϞʔςϜͷ෼ੳ
    
    
    ҰఆͷपظͰϥϯΩϯάʢτϦΨʔ, ࠜຊݪҼ, MTTR ౳ʣΛग़ͯ͠τϨϯυ֬ೝ΍஫ҙשىΛ͢Δ
    

    View Slide

21. Postmortem : ࢀߟυΩϡϝϯτ
    ʮαΠτϦϥΠΞϏϦςΟϫʔΫϒοΫʯ10ষ ϙετϞʔςϜͷจԽɿࣦഊ͔Βͷֶͼ
    
    
    ྑ͘ͳ͍ϙετϞʔςϜ, ྑ͍ϙετϞʔςϜ
    
    
    ʮSRE αΠτϦϥΠΞΠϦςΟΤϯδχΞϦϯάʯͷςϯϓϨʔτվగ൛
    
    
    https://bit.ly/2kIhFNC
    
    
    GoogleɿPostmortem Checklist
    
    
    https://g.co/SiteReliabilityWorkbookMaterials
    
    
    PagerDutyɿIncident Respons > After an Incident > Postmortem Template
    
    
    https://response.pagerduty.com/after/post_mortem_template/
    

    View Slide

22. Incident Fire Drill
    

    View Slide

23. Incident Fire Drill
    ΠϯγσϯτFire Drillͱ͸ʁ
    
    
    FireDrillʹආ೉܇࿅/ফ๷܇࿅
    
    
    Πϯγσϯτͷൃੜʹඋ͑ͯߦ͏໛ٖ܇࿅
    
    
    ΠϯγσϯτFire Drillͷ໨త
    
    
    ܦݧΛ { ੵΜͰ, ࢥ͍ग़ͯ͠ } ΋Β͏
    
    
    ༗ࣄͷࡍʹରԠʹ໎Θͳ͍Α͏ʹ͢Δ
    
    
    ΠϯγσϯτରԠͷҰ࿈ͷϑϩʔΛ఺ݕ͠վળ͢Δ
    

    View Slide

24. Incident Fire Drill : ૝ఆγφϦΦ
    ૝ఆγφϦΦΛߟ͑ΔʢAWSدΓʣ
    2. ௕࣌ؒ෮چͷݟࠐΈཱ͕ͨͳ͍AZো֐͕ൃੜͨ͠
    
    
    IaCͰյͯ͠΋ྑ͍؀ڥΛ࡞Γ (※)AWS FIS ౳Λ࢖ͬͯAZো֐Λىͯ͜͠ΈΔ (※)AWS Fault Injection Simulator
    
    
    ૝ఆ௨Γͷಈ࡞Λ͠ͳ͔ͬͨ৔߹ɺϚϧνAZରԠͷϥϯχϯάίετɾ޻਺͕ݟੵ΋ΕΔ
    
    
    AWS Resilience HubΛ࢖͏ͱɺܭըͨ͠RTO/RPOʹԊ͍ͬͯΔ͔ࣄલʹධՁͰ͖Δ
    1. ͱ͋Δमਖ਼ͷσϓϩΠޙɺػೳ͕ಈ࡞͠ͳ͘ͳͬͨ
    
    
    ϩά/ΦϒβʔόϏϦςΟπʔϧ౳Λ࢖ͬͨτϥϒϧγϡʔςΟϯάͷ܇࿅ɺ੾Γ໭͠ͷ܇࿅
    3. ৘ใ࿙Ӯ/ηΩϡϦςΟܥΠϯγσϯτ
    
    
    ηΩϡϦςΟΠϯγσϯτൃੜ࣌Λ૝ఆͨ͠܇࿅ → AWS GameDay / ײ૝هࣄ
    
    
    SIEM on AOSͷར׆༻ɺൃݟత౷੍ɾ༧๷త౷੍ͷ؍఺Ͱඞཁͳ४උ͕Ͱ͖͍ͯΔ͔Λݟ௚͢
    
    
    AWSηΩϡϦςΟΠϯγσϯτٙࣅମݧ GameDay ʹSREνʔϜͰࢀՃ͠ɺͱͯ΋ྑ͍ܦݧΛಘΒΕͨ
    

    View Slide

25. Incident Fire Drill : ࣮ࢪํ๏
    ࣮ࢪํ๏ͷྫ
    
    
    “ͱ͋Δमਖ਼ΛσϓϩΠޙɺػೳ͕ಈ࡞͠ͳ͘ͳͬͨ” ͱԾఆ
    
    
    SRE͕ओ࠵ɺ֤DEVνʔϜ͕ࢀՃऀ
    
    
    τʔλϧ60෼ɿ໨త/֓ཁ/࣭ٙԠ౴ 20෼, ࣮ࢪ30෼, ৼΓฦΓ10෼
    
    
    ΠϯγσϯτରԠϑϩʔͷҰ࿈ͷྲྀΕΛ௨ͯ͠΋Β͏
    
    
    ·ͣSEV൑ఆɺΤεΧϨʔγϣϯɺௐࠪ/ରԠɺϙετϞʔςϜ
    
    
    ࣮ࢪͷίπ
    
    
    VIDEOձٞͷϒϨΠΫΞ΢τϧʔϜΛ࢖ͬͯɺ֤νʔϜ͝ͱʹ࣮ࢪ
    
    
    SRE͸࢓੾Δͱ܇࿅ʹͳΒͳ͍ͷͰɺαϙʔτʹప͢Δ
    
    
    ࣭໰͕͋ͬͨΒճ౴͢Δఔ౓ͰOK
    

    View Slide

26. Incident Fire Drill : ఺ݕ಺༰
    ఺ݕ಺༰
    
    
    Ұ࿈ͷΠϯγσϯτରԠϑϩʔ͕௨ͤΔ͜ͱ
    
    
    ಺༰ʹෆඋ΍վળ఺͕ͳ͍͔
    
    
    ୭Ͱ΋SEV൑ఆ͕Ͱ͖Δ͔ɺ൑அʹ໎͏Օॴ͕ͳ͍͔
    
    
    υΩϡϝϯτ΍ΤεΧϨʔγϣϯઌΛ໎Θͣʹ୳ͤΔ͔
    
    
    ݪҼͷௐࠪ/෮چ͕Ͱ͖Δ͜ͱ
    
    
    ϩά, ΦϒβʔόϏϦςΟπʔϧ౳͔ΒΤϥʔ಺༰͕֬ೝͰ͖Δ͔
    
    
    ௐࠪ/෮چͷखॱ͕ڞ༗͞Ε͍ͯΔ͔
    
    
    ઀ଓ৘ใ΍ϊ΢ϋ΢΍ଐਓԽ͍ͯ͠ͳ͍͔
    
    
    ඞཁʹԠͯ͡ɺΠϯγσϯτରԠͷڞ༗ΛΦϯϘʔσΟϯάʹՃ͑ͯ΋Β͏
    

    View Slide

27. Incident Fire Drill : ৼΓฦΓ
    ৼΓฦΓ
    
    
    ֤νʔϜͷ୅දऀʹ ֶͼ/վળ఺ ʹ͍ͭͯҰݴͣͭίϝϯτΛ΋Β͏
    
    
    ଞνʔϜͷؾ෇͖͕ɺ
    
    
    શһʹΞϯέʔτΛ͓ئ͍͠ɺ࣍ճͷࢀߟʹ͢Δ
    
    
    Ξϯέʔτ݁ՌɿࢀՃऀͷ੠ʢൈਮʣ
    
    
    νʔϜ಺ͰΠϯγσϯτͷରԠखॱΛڞ༗ɾ֬ೝͰ͖ͯΑ͔ͬͨ
    
    
    υΩϡϝϯτ΍ϑϩʔͷվળ఺͕ݟ͔ͭͬͨ
    
    
    γφϦΦΛม͑ͯఆظతʹ࣮ࢪ͍ͨ͠
    
    
    ͕࣌ؒ଍Γͳ͔ͬͨ(આ໌20෼, ࣮ࢪ30෼, ৼΓฦΓ10෼ ӡӦଆ΋վળ͕ඞཁ)
    ࢀՃશνʔϜʹ
    
    
    ԿΒ͔ͷվળ఺͕ݟ͔ͭΓ·ͨ͠
    

    View Slide

28. ·ͱΊ
    

    View Slide

29. ·ͱΊ
    ޮ཰తͳΠϯγσϯτରԠΛ͢Δʹ͸ʁ
    
    
    ͍͟ͱ͍͏ͱ͖ରԠʹ໎Θ͵Α͏ɺ
    
    
    υΩϡϝϯτ΍ϑϩʔͷࣄલ੔උ͕ॏཁ
    
    
    ͦͯ͠ɺ๨Εͨࠒʹ΍ͬͯ͘ΔΠϯγσϯτʹඋ͑ͯɺ
    
    
    ఆظతʹτϨʔχϯά΍ݟ௚͠Λ͠·͠ΐ͏
    උ͑͋Ε͹ױ͍ͳ͠ʂ
    ຊηογϣϯ͕ɺօ༷ʹͱͬͯ
    
    
    ޮ཰తͳΠϯγσϯτରԠΛਐΊΔ͖͔͚ͬʹͳΕ͹޾͍Ͱ͢
    

    View Slide

30. View Slide