Share
Embed
Creating Awesome Change in SmartNews
DevOpsDaysTokyo 2025の登壇資料です。
https://confengine.com/conferences/devopsdays-tokyo-2025/proposal/21885/creating-awesome-change-in-smartnews-act
Ikuo Suyama
April 14, 2025
Transcript
103 slides
-
Who am I? / ͓·ͩΕ Ikuo Suyama / ಃࢁҭஉ •
Staff Engineer • Ads Backend Expert • Nov. 2020~ SmartNews, Inc. • Interest: Fishing, Camping, Gunpla, Anime -
5 ࠓ͓͢͠Δ͜ͱ 1. ΠϯγσϯτରԠͷݱͰಘΒΕͨܦݧͱݟ 2. Πϯγσϯτͷੳͱɺ͔ͦ͜Βߦͬͨରࡦ 3. ৫ʹ౷ҰϓϩηεΛਁಁͤ͞Δࢪࡦ Disclaimer 1:
ڧίϯςΩετґଘɺN=1ͷܦݧஊʂ Πϯγσϯτͱઓ͏ಛνʔϜͰͷؒͷܦݧ -
6 ࠓ͓͠͠ͳ͍ʢͰ͖ͳ͍ʣ͜ͱ 1.DevνʔϜͱOpsνʔϜͷڠۀ → DevνʔϜ͕ӡ༻োରԠ͍ͬͯΔલఏ 2.SRE, DevOps ͷ”ϕετϓϥΫςΟε”ͷదԠ → ͋͘·ͰݱͰಘΒΕͨܙͱܦݧ
Disclaimer 2: SREͷϓϩͰɺDevOpsͷϓϩͰ͋Γ·ͤΜ -
1.࢝ಈ: Assemble! ಛघ෦ୂ “ACT”! 2.ॳಈ: “Get our hands dirty”! 3.༂ਐ:
Incident Λʹ͢Δ!? 4.ؼؐ: ͞Εͨ՝ͱ͜Ε͔Β Agenda -
9 1-1. ࢝·Γ ͋Ε݄̕ͷ͜ͱͩͬͨ… Incident ͕ଟ͗͢Δʂ Incident Λݮ͢Δ λεΫϑΥʔεΛ࡞Δͧʂ Awesome
Change Team “ACT” ͩʂ CTO -
12 1-1. ࢝·Γ • CTO) Πϯγσϯτຊʹ “ଟ͍” ͷ͔ʁ • ͦͦΠϯγσϯτ͕ଟ͍ͬͯͲ͏ఆٛ͢Δʁ
• Ikuo) มߋຊʹଟ͍ͷ͔ʁ • ͦͦΠϯγσϯτ͕ଟ͍ݪҼมߋͳͷ͔ʁ • มߋͱԿͷมߋͳͷ͔ʁ ͜ͷ࣌Ͱํࠜڌͷͳ͍ɺ”Χϯ” ͪΐͬͱ͍ͯʂ ※ͨͩ͠γχΞΤϯδχΞͷᄿ֮Εͳ͍ -
14 1-2. ࠷ڧνʔϜΛूΊΔ ֤Division͔ΒΤʔεୡ͕ू·Δ…ʂ Ads News Ranking Push Notification Core
System (Infra) Mobile SmartView (Article) -
15 1-2. ࠷ڧνʔϜΛूΊΔ ֤Division͔ΒΤʔεୡ͕ू·Δ…ʂ Ads News Ranking Push Notification Core
System (Infra) Mobile SmartView (Article) Ads Ikuo! News & Push D! Ranking R! CoreSystem T! Mobile M! SmartView T! VPoE K! ※ ͷ߹্ࣗͷ͜ͱΤʔεͱݺ͍͓ͤͯͩ͘͞ئ͍͠·͢ (Manager) CTO Report To -
16 1-2. ࠷ڧνʔϜΛूΊΔ ֤Division͔ΒΤʔεୡ͕ू·Δ…ʂ Ads News Ranking Push Notification Core
System (Infra) Mobile SmartView (Article) Ads Ikuo! News & Push D! Ranking R! CoreSystem T! Mobile M! SmartView T! ※ ͷ߹্ࣗͷ͜ͱΤʔεͱݺ͍͓ͤͯͩ͘͞ئ͍͠·͢ VPoE K! (Manager) CTO Report To SREŧŔŕŪũƄŝſ -
18 1-3. νʔϜΛํ͚ͮΔ ᐆດͰ͕͑ͳ͍ͷઓ • “ΠϯγσϯτΛݮΒ͢” ͱ͍͏Ұݟγϯϓϧ͕ͩେ ͳۭؒ • Ͳ͔͜ΒखΛ͚Δ͔ʁ
• Կ͕ͰͲΜͳରࡦ͕༗ޮͳͷ͔ʁ • ͦͦΠϯγσϯτଟ͍ͷʢry -
19 1-3. νʔϜΛํ͚ͮΔ ໌֬ͳΰʔϧઃఆ • “Awesome Change” ͱ • ΫϦςΟΧϧͳΠϯγσϯτΛݮΒ͢
• SREϕετϓϥΫςΟεΛ৫ʹΠϯετʔϧ͢Δ • վળରKPI: • Mean Time Between Failure(MTBF) / Change Failure Rate(CFR) ʹΠϯγσϯτ • Mean Time to Recover(MTTR) ʹΠϯγσϯτղܾ࣌ؒ “զʑͳͥ͜͜ʹ͍Δͷ͔” ͷݴޠԽʂ VPoE͕͏·ͬͯ͘͘Ε·ͨ͠ -
20 1-3. νʔϜΛํ͚ͮΔ ໌֬ͳϓϥΠΦϦςΟઃఆ • P0: ΠϯγσϯτϋϯυϦϯάΛαϙʔτ͢Δ • P1: ΫϦςΟΧϧ͔ͭফԽ͞Ε͍ͯͳ͍ΠϯγσϯτΞ
ΫγϣϯΞΠςϜΛ௵͢ • P2: ΠϯγσϯτൃੜΛ͙ࠜຊతͳγεςϜվળ ؟ͷલΔ͜ͱ໌֬ʂ -
22 2-1. P0: ΠϯγσϯτϋϯυϦϯάͷαϙʔτ Get our Hands Dirtyɿͯ͢ͷΠϯγσϯτʹհೖ͢Δʂ • Πϯγσϯτ͕ى͜ΔͱɺͱΓ͋͑ͣACTϝϯόʔͷͩΕ͔ͷ
PagerDuty͕໐Δ • ݁ہACTશһΛΠϯγσϯτ͕ى͍ͬͯ͜Δͱ͜Ζʹট͢Δ • ࣗͷग़υϝΠϯͰ͋ΕফՐ׆ಈʹࢀՃ͢Δ • ͦ͏Ͱͳͯ͘ɺεςʔλεΞοϓσʔτඞཁͳਓࡐͷ֬อɺ Ϗδωεͱͷ࿈བྷͳͲΛങͬͯग़Δ ΩπΠ!! -
23 2-1. P0: ΠϯγσϯτϋϯυϦϯάͷαϙʔτ ΞϯνύλʔϯɿPager Monkey l43&͕ͯ͢ͷΦϯίʔϧΛॲཧ͢Δɺͭ·Γ൴Βࠓɺ αʔϏε͕μϯͨ͠ޕલ࣌ʹεΫϦϓτʹै͏͜ͱ͕ࣄͷɺ ϖʔδϟʔϞϯΩʔͱͳΔz 43&Λ࢝ΊΑ͏ষ43&ͷจԽ
વ Bad Practice ͯ͢ͷΠϯγσϯτʹհೖͯ͠Πϯγσϯτࢭ·Βͳ͍… -
24 2-1. P0: ΠϯγσϯτϋϯυϦϯάͷαϙʔτ Ξϯνύλʔϯ… ͚ͩͲɺѱ͍͜ͱ͔Γ͡Όͳ͍ Πϯγσϯτ = ACTͱ͍͏ୈҰى ͦͯ͠৴པஷۚΛಘͨʂ
ACTΠϯγσϯτʹհೖ͢Δ Πϯγσϯτͷͱ͖ACT͕ॿ͚ͯ͘ΕΔ ACTࣄͯ͠Δʂʂ -
25 2-2. P1:ΞΫγϣϯΞΠςϜΛ௵͢ ΕڈΒΕͨΞΫγϣϯΞΠςϜͨͪ ؒҧ͍ͳ͘ରԠ͞ΕͣɺΕڈΒΕ͍ͯΔͭΒ͕͍Δ • ͱͱΠϯγσϯτϨϙʔτΛ͢จԽ͕͋ͬͨ • ࠶ൃࢭͷΞΫγϣϯΞΠςϜهࡌ͞Ε͍ͯͨ •
͢Β͍͠ʂʂ • ͕ɺΞΫγϣϯΞΠςϜཧ͞Ε͍ͯͳ͔ͬͨ • ୲ɺظݶɺྃεςʔλε • !!??!??! • ͔ͭɺϨϙʔτͷϑΥʔϚοτDivision͝ͱʹҟͳͬͨ • ͳΜͳΒ୲ऀ͝ͱʹҟͳͬͨ -
27 2-2. P1:ΞΫγϣϯΞΠςϜΛ௵͢ Get our Hands DirtyɿखಈͰσʔλཧ ڭ܇̍: σʔλͳΔ͘ ػցॲཧͰ͖ΔϑΥʔϚοτͰͤʂʂ
ͤʂաڈ̍ͷΠϯγσϯτϨϙʔτͷAI શ෦खಈͰNotion DatabaseʹҠߦͨ͠Ζʂ ※Databaseʹͯ͠͠·͑APIܦ༝Ͱσʔλ͕औΕΔͷͰͲ͏ʹͰͳΔ ڭ܇̎ɿతͷͨΊʹటष͍खஈΛऔΔ͜ͱΛԀ͏ͳʂʂ Get our Hands Dirty -
28 2-2. P1:ΞΫγϣϯΞΠςϜΛ௵͢ Get our Hands DirtyɿऴΘͬͯͳ͍ͭΛͻͨ͢ΒରԠ͢Δ ͜ΕऴΘͬͱΒΜΜ͚ʂ ௵ͤʂʂʂ ڭ܇̏:ΈΜͳ͍͠ɻཔΜͰΒͳ͍ͷͰͳ͘Εͳ͍
ࣗͨͪͷखΛಈ͔ͤʂʂ Get our Hands Dirty -
29 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ • Division͕ҧ͑ΠϯγσϯτରԠͷΓํɺྲّྀҟ ͳΔ • աڈʹશࣾ౷ҰͷϓϩτίϧΛ࡞ΔࢼΈ͕͋ͬͨ
• ”Incident Response Framework:IRF” • ͕ɺਁಁ/ར༻͍ͯ͠ͳ͔ͬͨ • ಛఆͷDivisionͷཁ͔݅͠ߟྀ͞Εͯͳ͔ͬͨʂ ͦͦͳͥશࣾͰ౷Ұ͞Εͨϓϩηε͕ͳ͔ͬͨʁ -
30 • IRFࣗମϓϩηεͱͯ͠Α͘Ͱ͖͍ͯͨ • ͜ΕΛϕʔεʹɺ • શࣾͰ౷ҰͰ͖Δखॱ … ֤Τʔε͔ΒͷυϝΠϯࣝͱܦݧͷೖ •
͔ͭܰྔͳͷ • Πϯγσϯτͷ࠷தʹෳࡶͳखॱΛҾͬுΔʂ • ެ։͞Ε͍ͯΔଞࣾͷFWࢀߟʹɺྑ͍ͱ͜ΖΛऔΓೖΕͨ • e.g. Pager Duty Incident Response Ͳ͏ͬͯશࣾ౷ҰϓϩηεɺϑϨʔϜϫʔΫΛ࡞Δʁ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ ͯ͢ͷྖҬΛΧόʔͨ͠νʔϜ͔ͩͬͨΒͦ͜Մೳ -
31 IRF 2.0 Contents 1. Role, Playbook 2. Severity Definition
3. Workflow 4. Communication Guideline 5. Incident Report Template, Postmortem ॏཁͳͱ͜ΖΛհ͠·͢ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ ৄࡉεϥΠυΛ֬͝ೝ͍ͩ͘͞ʂ -
32 IRF 2.0: Role, Playbook • On-Call Engineer • ΦϯίʔϧΛड͚ΔΤϯδχΞɻΞϥʔτͷτϦΞʔδΛߦ͍ɺඞཁͰ͋ΕICʹΤε
ΧϨʔγϣϯͯ͠IRFΛ։࢝͢Δ(IncidentΛએݴ͢Δ)ɻ • Incident Commander(IC) • ΠϯγσϯτରԠͷࢦشΛͱΔਓɻඞཁͳਓΛूΊɺใΛཧ͢Δɻ֎෦ͱͷίϛϡ χέʔγϣϯʢCLʣΛ݉͢Δ͜ͱ͋Δɻ௨ৗTech Lead/Engineering Managerɻ • ࣮ࡍͷՐফ͠࡞ۀͰͳ͘ɺใɾঢ়گཧͱஅ͕ • Responder • ࣮ࡍͷՐফ͠࡞ۀʢϩʔϧόοΫઃఆมߋʣΛߦ͏ɻ • Communication Lead(CL) • ֎෦εςʔΫϗϧμʔʢ͜͜ͰΤϯδχΞҎ֎ʣͱͷίϛϡχέʔγϣϯΛ୲͢ Δɻ ICͱResponderͷ͕ΩϞ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ -
33 IRF 2.0: Severity Definition IC͕Πϯγσϯτએݴ࣌ʹఆతʹܾఆ͢Δɻ࠷ऴධՁϙετϞʔςϜͰܾ ·Δ • 🔥 SEV-1
• χϡʔεߪಡͳͲίΞUXػೳ͕શఀࢭ • 🧨 SEV-2 • ίΞUXػೳͷҰ෦ఀࢭɺαϒUXػೳͷશఀࢭ • 🕯 SEV-3 • αϒUXػೳͷҰ෦ఀࢭ ॳಈͷ࣌ͰSEVʹͨΓΛ͚͓ͭͯ͘͜ͱ͕؊ཁɻ ʢγϏΞͳΠϯγσϯτΑΓૣ͘ղܾ͍ͨ͠ʣ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ -
34 IRF 2.0: Workflow Πϯγσϯτ͕࢝·͔ͬͯΒऴΘΔ·ͰͷྲྀΕɻ🩸 : Bleeding, ग़݂தͷεςʔλε 1. 🩸
Occurrence/ൃੜ • ͱͳΔࣄͷൃੜɻσϓϩΠઃఆมߋͳͲ͕τϦΨʔ 2. 🩸 Detection/ݕ • ΞϥʔτͳͲʹΑΓɺOnCaller͕Λݕͨ͠ঢ়ଶɻτϦΞʔδΛ։࢝ 3. 🩸 Declaration/એݴ • Πϯγσϯτͷ ”એݴ”ɻIRFʹଇΓɺICͷࢦشͷͱࢭ݂ରԠ։࢝ • ಉ࣌ʹඞཁͳ֎෦ίϛϡχέʔγϣϯΛ։࢝ɻग़݂தܧଓతͳΞοϓσʔτ 4. ❤🩹 Mitigation/؇ • มߋͷϩʔϧόοΫͳͲͰҰ࣌ݪҼΛഉআɺඃͷ֦ࢄΛఀࢭ 5. Resolution/ղܾ • ෆ۩߹ͷमਖ਼σʔλิਖ਼ͳͲɺ߃ٱରԠͷྃɻશࢭ݂ 6. Postmortem/ࣄޙੳ • ΠϯγσϯτϨϙʔτΛݩʹɺࠜຊݪҼͷٹ໋ͱ࠶ൃࢭࡦͷݕ౼ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ -
35 IRF 2.0: Communication Guideline ίϛϡχέʔγϣϯʹ͏ॴʢSlackͷνϟϯωϧʣͷఆٛ • #incident • શମͷεςʔλεपɺ֎෦εςʔΫϗϧμʔͱͷίϛϡχέʔγϣϯ
• #incident-irf-[incidentId]-[title] • ղܾͷͨΊͷٕज़తͳίϛϡχέʔγϣϯɻؔ࿈͢Δใɾٞͯ͢ूΊΔ • ඞཁʹԠͯ͡WAR ROOM(Online, Google Meet)Λཱͯͯू߹ ٞใ͕̍ͭͷνϟϯωϧʹू·ͬͯΔͱϨϙʔτੜ࣌ʹศར 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ -
36 IRF 2.0: Incident Report Template & Postmortem શࣾ౷ҰϑΥʔϚοτͷదԠ •
Summary • Impact • Direct Cause, Mitigation • Root Cause Analysis (5-whys) • ݪҼͱࠜຊݪҼΛ͚ͯੳ͢Δ͜ͱ͕ॏཁʂ • ͜͜ʹରͯ͠ΞΫγϣϯΞΠςϜΛઃఆ͠ɺ࠶ൃࢭ • Action Items • Timeline • ػցॲཧͰ͖ΔϑΥʔϚοτͰ!!!! DivisionࣄʹҟͳͬͨςϯϓϨʔτΛ౷Ұʢ͍ͩ͡ʣɺPostmortemͷҰݩԽ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ -
39 Get our Hands Dirtyɿͯ͢ͷΠϯγσϯτʹհೖ͠ɺ ແཧΓIRF2.0ϓϩηεΛదԠ͢Δ Ͳ͏͜ΜʹͪIRF͓͡͞ΜͰ͢ ͡Ό͕͋ΠϯγσϯτίϚϯμʔΓ·͢Ͷʂ Έͳ͞ΜՐফ͠ʹूத͍ͯͩ͘͠͞ʂʂ ڭ܇̐:ۓٸ࣌ʹ৽͍͠ϓϩτίϧΛֶΜͰ͍Δ༨༟ͳͲͳ͍ʂ
࣮ફ͋ΔͷΈʂʂ 2-3. P2: ࠜຊվળ ౷ҰΠϯγσϯτରԠϓϩηεಋೖ ڭ܇̑:ࣗୡͰ·ͣ͑ʂϑΟʔυόοΫϧʔϓΛճͤʂ -
2-4. P2: ࠜຊվળ Πϯγσϯτͷղ૾ΛߴΊΔ 43 ॏཁͳؾ͖ͮɿKPI͕͑ͯͳ͍ʂ ͍ͯ͏͔ࠓ݄ Կ݅Πϯγσϯτॲཧͨ͠ʁ ઌ݄…ʁ Ұ݅ͲΕ͘Β͍Ͱ
ղܾͰ͖ͨʁ ʹ Θ ͔ Β ͳ ͍!! -
σʔλఆٛɿΠϯγσϯτΛϞσϦϯά͢Δ • Insidentͷଐੑ • Title • Status • State Machine.
ޙड़ • Severity • SEV 1~3(IRF2.0) • Direct Cause • ޙड़ • Direct Cause System • MicroServiceͷҙຯ୯ҐͰͷίϯϙʔωϯτ܈ • Direct Cause Workload • Online Service, Offline Pipeline, … ՄೳͳݶΓEnumΛఆٛ͢Δ 46 2-4. P2: ࠜຊվળ Πϯγσϯτͷղ૾ΛߴΊΔ ࣗ༝ೖྗͰΧʔσΟφϦςΟ͕ߴ͘ͳΓ͗ͯ͢·ͱͳੳ͕Ͱ͖ͳ͍ -
σʔλऩूɿΠϯγσϯτϨϙʔτ ઌʹఆٛͨ͠σʔλ߲ΛؚΊΔΑ͏ɺΠϯγσϯτϨϙʔτͷςϯ ϓϨʔτΛΞοϓσʔτ • ඞཁͳ߲ΛඞਢೖྗͷAttributeʹ • EventTimelineΛೖྗ͢ΔNotionDatabase ΛՃ • State͕มԽͨ࣌ؒ͠Λهͯ͠Β͏
• ػցॲཧͰ͖ΔϑΥʔϚοτͰ!!!!! σʔλఆ͕͔ٛͬ͠Γ͍ͯ͠ΕɺιʔεϑϨΩγϒϧ (ͪΖΜ৴པͰ͖ΔσʔλͰ͋Δલఏ) 49 2-4. P2: ࠜຊվળ Πϯγσϯτͷղ૾ΛߴΊΔ -
աڈσʔλΛͲ͏͢ΔʁͪΖΜGet our Hands Dirty! ͤʂաڈ̍ͷΠϯγσϯτϨϙʔτ શ෦खಈͰ৽ϑΥʔϚοτʹҠߦͨ͠Ζʂʂ Re) ڭ܇̎ɿతͷͨΊʹటष͍खஈΛऔΔ͜ͱΛԀ͏ͳʂʂ ̍ि͔͚ؒͯগͣͭ͠ɺνʔϜશһͰख͚ͯ͠ҠߦΛ 2-4.
P2: ࠜຊվળ Πϯγσϯτͷղ૾ΛߴΊΔ Get our Hands Dirty -
ॏཁͳࢦඪͷ؍ଌɿMTTRͷ؍ଌͱࡉԽ 1. Occurred 2. Detected 3. Declared 4. Mitigated 5.
Resolved Time To Detect Time To Mitigate Ͳ͜ʹ͕͔͔͍࣌ؒͬͯΔ͔ɺݱࡏ͕Θ͔ͬͨʂ Time To Resolve 2-4. P2: ࠜຊվળ Πϯγσϯτͷղ૾ΛߴΊΔ -
58 3-1. ΠϯγσϯτΛݮΒ͢ͱʁ ΠϯγσϯτʹΑΔΠϯύΫτ e.g. Revenue, Reputation, Effectiveness…. Ͱ͢ΑͶʁ ಛʹRevenue
Loss… ຊʹݮΒ͍ͨ͠ͷ ݮΒ͍ͨ͠ͷ Πϯγσϯτʁ🤔 -
59 3-1. ΠϯγσϯτΛݮΒ͢ͱʁ MTTD + MTTM (ࢭ݂ʹ͔͔ͬͨ࣌ؒ) ΠϯγσϯτͷΠϯύΫτΛݟੵΔ Severity Factor
(ΠϯγσϯτͷӨڹ߹͍) Πϯγσϯτ × Σ toC, ࠂϏδωεͩͱ͍͍ͩͨ͜ΕͰRevenueΠϯύΫτ͕ܾ·Δ • ͳΔ͍ͨ͘͘͠ • վળ͕ൺֱత༰қɺख͕͚͍ͭ͢ • ͳΔ͘େ͖ͳΠϯγσϯτΛݮΒ͍ͨ͠ • ͕ɺίϯτϩʔϧ͕͍͠ • ͳΔ͘ΛݮΒ͍ͨ͠ • தظͷ׆ಈ͕ඞཁ -
Σ 60 3-1. ΠϯγσϯτΛݮΒ͢ͱʁ MTTD + MTTM (ࢭ݂ʹ͔͔ͬͨ࣌ؒ) ΠϯγσϯτͷΠϯύΫτΛݟੵΔ Severity
Factor (ΠϯγσϯτͷӨڹ߹͍) Πϯγσϯτ × • ͳΔ͍ͨ͘͘͠ • վળ͕ൺֱత༰қɺख͕͚͍ͭ͢ • ͳΔ͘େ͖ͳΠϯγσϯτΛݮΒ͍ͨ͠ • ͕ɺίϯτϩʔϧ͕͍͠ • ͳΔ͘ΛݮΒ͍ͨ͠ • தظͷ׆ಈ͕ඞཁ ͔͜͜ΒखΛ͚͍ͯ͘ ACT݁ॳʹཱͯͨKPIͱϚον͍ͯ͠Δʂ ͕ɺϲ݄ͷܦݧͰղ૾͕ΑΓ্͕ͬͨ -
MTTRͷղ૾Λ্͛ΔɿεςʔτϚγϯ 1. Occurred 2. Detected 3. Declared 4. Mitigated 5.
Resolved Time To Detect Time To Resolve Time To Mitigate ͦΕͧΕॏཁɺରࡦ͕ҟͳΔʂ ݕʹ͔͔Δ࣌ؒɻ ओʹAlertingͷྖҬ ݕʙࢭ݂ʹ͔͔Δ࣌ؒɻ ࠷ΫϦςΟΧϧ͕ͩɺΞϓϩʔν͍͢͠ IRFͷྖҬ ࢭ݂ʙࠜຊରԠ/ิਖ਼ͳͲޙॲཧʹ͔͔Δ࣌ؒɻ ͢Ͱʹࢭ݂͞Ε͍ͯΔͷͰɺ͞ΑΓਖ਼֬͞ ͕ٻΊΒΕΔ 3-2. Πϯγσϯτղܾ࣌ؒͷΞϓϩʔν -
63 MTTD(Mean Time To Detect) ʹର͢ΔΞϓϩʔν Ξϥʔτͷඋ • ݕͰ͖ͳ͔ͬͨ/Εͨʹ৽͘͠ΞϥʔτΛ͚ͭΑ͏ɺ͏·͍͔͘ͳ͍ •
“over-monitoring is a harder problem to solve than under-monitoring.” — SRE: How Google Runs Production System • False Positive Ξϥʔτଟ͗͢ɺຒΕΔ • SLO + Error Budget ʹΑΔΞϥʔτͷγϑτ • Pager͞ΕͨΒΠϯγσϯτɺ͕ཧ • ҰேҰ༦ʹ͍͔ͳ͍ʂ ͞Εͨ՝ɻ̐ষͰ͓͠·͢ʂ 3-2. Πϯγσϯτղܾ࣌ؒͷΞϓϩʔν -
64 MTTM(Mean Time To Mitigate) ʹର͢ΔΞϓϩʔν ౷ҰFWɿIRF2.0 • Πϯγσϯτͷج४ͷ໌ࣔԽ •
ରԠϑϩʔɾίϛϡχέʔγϣϯΨΠυϥΠϯͷ౷Ұ • Responder / Commander ͷ • ٴͼτϨʔχϯάɺΤʔεୡ͕എதΛݟͤΔ ΤʔεೖͱIRF2.0ͷਁಁ͕ޮՌ͖ͯΊΜʂ 3-2. Πϯγσϯτղܾ࣌ؒͷΞϓϩʔν -
69 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̍ɿຊ൪ೖલͷςετʁ PostMortemʹͯ… • Why. ͳͥςετͤͣʹຊ൪ʹೖ͞ΕΔͷͰ͔͢ʁ • ຊ൪Ͱ͔͠ςετͰ͖ͳ͍͔ΒͰ͢
• Why. ͳͥຊ൪Ͱ͔͠ςετͰ͖ͳ͍ͷͰ͔͢ʁ • σʔλෆɺStagingςετڥͷෆඋ • …. ϤγʂStagingڥΛඋ͢Δͧʂ -
70 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̍ɿStagingڥඋ ಓɿ૾ͷ10ഒେมʂ • ίϯϙʔωϯτ͕ࢮ͵΄Ͳ͋Δ • News, Ads,
InfraͱDivisionຖʹҟͳΔཁٻɺར༻ํ๏ • AdstoB, ͓ۚʹ݁ʂ͔ͬ͠Γ͔ͬͪΓ • NewstoC, ػೳఏڙεϐʔυ༏ઌʂ ͱΓ͋͑ͣͯ͢STGඋ͠·͢ɺແཧͩ͠ҙຯ͕ͳͦ͞͏ɻ ൺֱతཁ͕େ͖͍Ads͔ΒରԠத -
72 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̎ɿUnitTestͷੳ • ςετΧόϨοδΛऩूͯ͠ͳ͍ γεςϜɾίϯϙʔωϯτʹಥܸ ͠ɺΧόϨοδΛग़͢PRΛྔ࢈ • γεςϜ͝ͱʹUTΧόϨοδ
ͱɺোൃੜΛϓϩοτ # of Incident Ave. Coverage -
73 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̎ɿUnitTestͷੳ • ςετΧόϨοδͱো૬ؔ͢Δͷ͔ʁ → ૬ؔग़ͨɻ • ͕ɺςετΧόϨοδΛ্͛Εো͕ݮΔ͔ʁΘ͔Βͳ͍
ʢҼՌͰͳ͍ʣ • ͔͠͠υϝΠϯࣝΛͬͯɺγεςϜ/νʔϜ୯ҐͰݟͯΈΔͱɺ ͔֬ʹΧόϨοδ͕͘ɾΠϯγσϯτ͕ଟ͍ͱ͜Ζཧ༝͕͋Γ ͦ͏ • UTΛ࣮ͣ͠Β͍/UTΛ࣮͢ΔจԽ͕ͳ͍ etcetc… Ϥγʂͱʹ͔͘UT͕গͳ͍ͱ͜Ζʹಥܸͯ͠ UTΛ࣮͠·͘Δͧʂʂ -
74 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̎ɿUnitTestͷඋ Get our Hands Dirtyɿย͔ͬΒUTΛ͚ͭΔ 2. Sonarqube
Ͱߦ͕ଟ͘ɺCoverage͕͍ϑΝΠϧΛݟ͚ͭΔ 3. LLMͷྗΛआΓͯUTΛ࣮͠·͘Δ 4. ίϯϙʔωϯτશମͰ> 50% ʹͳΔ·Ͱ܁Γฦ͠ ̏ʙ̐ίϯϙʔωϯτ͕ͬͨম͚ੴʹਫ αϯϓϧ͕͋Εɺޙͬͯ͘ΕΔͩΖ͏ͱࢥ͍ͬͯͨ… -
75 3-3. ΠϯγσϯτͷΞϓϩʔν ςετෆʹର͢ΔΞϓϩʔν̎ɿUnitTestͷඋ • LLMͰࣗಈੜͨ͠Βʁ • ͍·ͷͱ͜Ζਫ਼͕͍·͍ͪ • ͦͦUTΛܧଓతʹ࣮͢Δश׳͕νʔϜʹඞཁ
• ͕ɺͦ͏͢ΔΠϯηϯςΟϒɺՁײ͕ແ͍ • ೲظʹΘΕ͍ͯͯɺUTʹׂ͕࣌ؒ͘ͳ͍(!) ৫ɺจԽͷΞϓϩʔν͕ඞཁʂ ͞Εͨ՝ɻ4ষଓ͘… -
77 3-3. ΠϯγσϯτͷΞϓϩʔν ઃఆมߋʹର͢ΔΞϓϩʔν • ओͳ”ઃఆมߋ” • ΦϯϥΠϯͰΞϓϦέʔγϣϯͷڍಈΛ੍ޚ͢Δػߏ • A/BςετɺFeature
Flag • ͲͪΒಠ࣮ࣗͷϓϥοτϑΥʔϜΛ͕࣋ͭɺෳࡶ • ҙਤ͠ͳ͍ϢʔβʔͷA/BదԠɺޡͬͨઃఆʹΑΔ ͕ଟൃ ϤγʂA/BςετͱϑΟʔνϟʔϑϥάΛ උ͢Δͧʂʂ -
78 3-3. ΠϯγσϯτͷΞϓϩʔν ઃఆมߋʹର͢ΔΞϓϩʔν • ΘΕ͍ͯͳ͍ʢσϑΥϧτԽͨ͠ʣϑΟʔνϟϑϥά ͷҰ੪আ • ϑΟʔνϟʔϑϥάར༻ج४ͷࡦఆ •
όϦσʔγϣϯͷڧԽ • ʢύʔεΤϥʔʹͳΔઃఆ͕ೖྗͰ͖͍ͯͨ…ʣ AB ςετϓϥοτϑΥʔϜνʔϜͱڠۀ͠ɺ ϢʔβʔϏϦςΟؚΊେ෯ͳվળΛਪਐʂ -
80 3-3. ΠϯγσϯτͷΞϓϩʔν Offline Batch ʹର͢ΔΞϓϩʔν • ετϦʔϛϯάॲཧͳFilnkΦϑϥΠϯόον͕ଟ • Server
→ Kafka → Flink → Scylla, ClickHouse, … • ઐνʔϜʹΑΔಠࣗ։ൃϓϥοτϑΥʔϜ • ΞϓϦέʔγϣϯνʔϜʹFlinkΤΩεύʔτ͕গͳ͘ɺ ύϑΥʔϚϯε࠶ىಈ࣌ͷ͕ଟൃ ϤγʂFlink ϓϥοτϑΥʔϜΛ උ͢Δͧʂʂ -
81 3-3. ΠϯγσϯτͷΞϓϩʔν Offline Batch ʹର͢ΔΞϓϩʔν • ϓϥοτϑΥʔϜࣗମͷվળ • UIͷվળɺࣗಈσϓϩΠɺ…
• ϕετϓϥΫςΟεͷڭ • υΩϡϝϯτඋ • ςετΛؚΉςϯϓϨʔτϓϩδΣΫτͷެ։ • ֤ίϯϙʔωϯτʹϦϑΝΫλPRΛૹ • ϕετϓϥΫςΟεͱςετΛ࣮ ϓϥοτϑΥʔϜνʔϜʹڠྗΛڼ͗ɺ ϓϥοτϑΥʔϜͷվળͱυΩϡϝϯτඋΛ࣮ࢪʂ -
82 3-4. ݁ՌɿΠϯγσϯτ””ʹͳͬͨͷ͔!? ACT era vs Before ACT Πϯγσϯτ… +32%
૿Ճ!! MTTR… -48% ݮগ!! ݮ!!! -
83 3-4. ݁ՌɿΠϯγσϯτ””ʹͳͬͨͷ͔!? ACT era vs Before ACT Πϯγσϯτ… +32%
૿Ճ!! MTTR… -48% ݮগ!! ݮ!!! ͪ ΐ ͬ ͱ ͯ͌ ʂ -
84 3-4. ݁ՌɿΠϯγσϯτ””ʹͳͬͨͷ͔!? Πϯγσϯτ૿͑ͯ·͢Α…ʁ • قઅੑʁ݄͕̍̎ඈͼൈ͚ͯଟ͍ • ٳՋલͷΓࠐΈมߋࣄނʁ • IRF2.0ਁಁͷ෭࡞༻ʁ
• ΠϯγσϯτఆٛʹΑΔݕײ ͷ্ • ϚζϩʔͷϋϯϚʔ: “͠IRF͔͍࣋ͬͯ͠ ͳ͚Εɺ͕ͯ͢Πϯγσϯτʹݟ͑Δ” • ݄̍Ҏ߱ݮগ ܧଓతͳվળ׆ಈ͕ඞཁ -
85 3-4. ݁ՌɿΠϯγσϯτ””ʹͳͬͨͷ͔!? ҰํɺMTTRݮʂ • ಛʹ MTTMitigate ʹܶతͳվળ • IRF2.0ޮՌͱߟ͍͑ͯΔ
• ҰํMTTDetectେ͖ͳվળͳ͠ • Detection ࠓޙͷ՝ɻΞ ϥʔτվળʹऔΓΉ վળʹ͔֬ͳखԠ͑ʂ -
89 ͦͦΠϯγσϯτΛ̌ʹ͍ͨ͠ʢͰ͖Δʣͷ͔ʁ ݱ࣮తʹͲͪΒແཧ… • ΠϯγσϯτΛۃখԽ͢Δʹʁ • ϦϦʔεΛͳ͘͢ʁ • →؇͔ͳࢮ😇 •
ແݶʹίετʢϦιʔεɺ࣌ؒʣΛԼ͢Δʁ • ೖͨ͠ϦιʔεͱΠϯγσϯτൃੜ(͓ͦΒ͘)૬ؔ͢Δ • Αͬͯɺ100%҆શͱߟ͑ΒΕΔ·Ͱͻͨ͢Βςετ͢Δ 4-1. ͞Εͨ՝ɿϦεΫཧ, Ξϥʔτվળ -
90 ೲظͱ࣭ɺίετͱΠϯγσϯτͷόϥϯεΛऔΓ͍ͨ • ͕ɺγεςϜɾϓϩδΣΫτ͝ͱʹόϥϯεҟͳΔ • ٻΊΒΕΔεϐʔυɺϦϦʔεස • ԼͰ͖Δίετ • ڐ༰Ͱ͖ΔϦεΫʢ㲈Πϯγσϯτɺมߋࣦഊʣ
• ྫɿ • AdstoB, ͓ۚʹ݁ʂ͔ͬ͠Γ͔ͬͪΓ • NewstoC, ػೳఏڙεϐʔυ༏ઌʂ ʹϦεΫڐ༰ΛԽ͠ɺ ΠϯγσϯτΛίϯτʔϧ͍ͨ͠ 4-1. ͞Εͨ՝ɿϦεΫཧ, Ξϥʔτվળ -
91 ϦεΫڐ༰ͷ໌ࣔԽɿSLOͱError Budget • SLO = αʔϏεϨϕϧඪ ~ ͲΕ͘Β͍Τϥʔ͕ڐ͞ΕΔ͔ʁ •
e.g. 99.9% available -> 0.1%ڐ༰͞ΕΔ • ࣮ࡍʹUXʹة͕͋ΔSLI(Indicator)ʹObjective(ඪ)Λ͚ͭΔ • Error Budget = ڐ༰Ͱ͖ΔΤϥʔ͕͋ͱͲΕ͘Β͍͍ͬͯΔ͔ • Error Budget ͕͍ͬͯΔ = ΞΫηϧΛ౿ΊΔ • ଟগແͳϦϦʔεڐ༰Ͱ͖Δ • Error Budget ͕ރׇͨ͠ = ڐ༰Ͱ͖ͳ͍ϨϕϧͷUXͷᆝଛ • ͜ΕҎ্ϦεΫΛऔ͍͚ͬͯͳ͍ɻεϐʔυμϯ — Ref: Implementing SLOs — Google SRE Error BudgetʹΑͬͯϦεΫڐ༰ΛදݱͰ͖Δ ཧతʹྑͦ͞͏ 4-1. ͞Εͨ՝ɿϦεΫཧ, Ξϥʔτվળ -
92 Ξϥʔτͷվળ ʔ Ξϥʔτ = Πϯγσϯτ ϤγʂSLO Λඋ͢Δͧʂʂ • Error
Budget ͷফඅ, Burn Rate ʹΑͬͯΞϥʔτ͢Δ • ٸͳ Error Budget ফඅΛΞϥʔτ • ์ஔ͢Δͱ༧ࢉ͕ރׇ͢Δɻͭ·ΓSLOʹҧ͢Δ • →࣮ࡍͷUXʹة͕͋Δʂ • →์ஔ͍͚ͯ͠ͳ͍ʂʹΠϯγσϯτ — Ref: Alerting on SLOs 4-1. ͞Εͨ՝ɿϦεΫཧ, Ξϥʔτվળ ྑͦ͞͏ -
93 4-2. ͞Εͨ՝ɿ৫ͱจԽͷΞϓϩʔν Ұ෦ʹSLOΛಋೖͯ͠Έ͕ͨ… • ࣮ޮੑͷ͋ΔSLOΛఆٛɾ࣮͢Δͷ؆୯͡Όͳ͍ • ϏδωεɾPdMʹ͖͍ͯ͑Λ͍࣋ͬͯͳ͍ • ࣮ͦͦʹ͔͚Δ࣌ؒΛ೧ग़Ͱ͖ͳ͍
• ςετͷ࣌ؒ͢Β೧ग़Ͱ͖ͳ͍ͷʹ… • Α͠Μ࣮Ͱ͖ͨͱͯɺ९क͞Εͳ͚Εҙຯ͕ͳ͍ ۜͷؙͳͲͳ͍… -
94 4-2. ͞Εͨ՝ɿ৫ͱจԽͷΞϓϩʔν SLO Λػೳͤ͞Δʹʁ • ৫શମͷཧղͱڠྗ͕ෆՄܽ • ΤϯδχΞ͚ͩͰͳ͘ɺϏδωεɺPdMר͖ࠐΉඞཁੑ •
จԽͷΞϓϩʔν͕ඞཁ • ڀۃతʹͳʹΛՁͱ͢Δ͔ɺͱ͍͏ • SLOͰίετͱϦεΫͷόϥϯεΛऔΔ͜ͱΛՁΛ৴ ͡ɺ࣮ߦͰ͖Δ͔ ΤϯδχΞϦϯάจԽʹSLOɺ ͻ͍ͯSREΛΠϯετʔϧ͍ͨ͠ -
95 4-2. ͞Εͨ՝ɿ৫ͱจԽͷΞϓϩʔν SLO Λػೳͤ͞Δʹʁ • Ͳ͏Ξϓϩʔν͢Δ͔ • BottomUp: •
Eng, Biz, PdM ΒεςʔΫϗϧμʔͷڭɺτ Ϩʔχϯά • TopDown: • ্෦͔Βͷࢧ࣋ɺࢦࣔ SRE, DevOpsจԽɺҰʹͯ͠Βͣ Λ͔͘ಓ͔ͭܧଓతͳ׆ಈ͕ඞཁ -
96 4-3. ͜Ε͔Β… ACTؒͷظݶ͖ظ͕ऴΘΖ͏ͱ͍ͯͨ͠ • ՝: SNͷΤϯδχΞϦϯάจԽʹSREΛΠϯετʔϧ͢Δ • SLO ͷ࣮ɺ९क
• ଞʹ… • Observability ͷ্ • DORA Metrics ͷऩूٴͼࢹɺ९क, etcetc… → ܧଓతͳ׆ಈ͕ඞཁ ͜͜·ͰͷาΈΛࢭΊͣɺ ACTղࢄޙ͞Εͨ՝ʹཱ͔ͪ͏ʹʁ -
98 4-2. ͜Ε͔Β… ACTΛͲ͏ղࢄ͢Δ͔ʁ — νʔϜͷ͑ • ٫Լʂ • ΈΜͳSREΛϑϧλΠϜδϣϒʹ͍ͨ͠Θ͚Ͱͳ͍
• “X%” ͷ࣌ؒΛΞϩέʔτ͢Δɺ͕ػೳͨͨ͠Ί͕͠ͳ͍ • ݁ • Ex-ACTorࠓޙSREͷܒखॿ͚Λߦ͏͕ɺ͕࣌ؒ ͔͔ͬͯઐͷSREνʔϜΛ্ཱͪ͛Δɻ νʔϜͰܾٞͯ͠ΊΒΕͨɻ Γͨ͜͠ͱଟʑ͋Δ͕ޙչͳ͍ʂ -
99 4-2. ͜Ε͔Β… ΅͘Βͷ Awesome Change! ACTͱͯ͠ͷؒͷʢΩπΠ!!ʣظऴΘͬͨ Awesome Change ͕࡞Ε͔ͨ…ਖ਼Θ͔Βͳ͍͚Ͳɺ
SREͱ͍͏ཱྀ͍ͷҰาΛ౿Έग़ͤͨɺ ͱ͍͏ײ৮͋Δʂ ͦͯ͠6ϲ݄Λઓ͍ൈ͍ͨνʔϜϝΠτʹײँʂʂ -
103 References • SREΛ͡ΊΑ͏―ݸਓͱ৫ʹΑΔ৴པੑ֫ಘͷୈҰา • SRE αΠτϦϥΠΞϏϦςΟΤϯδχΞϦϯά―Googleͷ৴པੑΛࢧ ͑ΔΤϯδχΞϦϯάνʔϜ • SRE
Google Workbook • Effective DevOps 4ຊபʹΑΔ࣋ଓՄೳͳ৫จԽͷҭͯํ • Fearless Change ΞδϟΠϧʹޮ͘ ΞΠσΞΛ৫ʹΊΔͨΊͷ48 ͷύλʔϯ