Share
Embed
Transcript
30 slides
-
ΧΦεΤϯδχΞϦϯάͲ͔͜Β ͖ͨͷ͔ɺͦͷઌʹԿ͕͋Δͷ͔ Web System Artchitecture ݚڀձ #4 @itkq Web System
Artchitecture ݚڀձ #4 (@itkq) 1 -
whoami • @itkq [ˈɪtəkəʊ] • SRE @ Cookpad • SLI/SLO,
Մ༻ੑ, ... • Web γεςϜΛࣦࣗͤͯ͞৬͍ͨ͠ Web System Artchitecture ݚڀձ #4 (@itkq) 2 -
ൃද༰ • ࣗݾհ • ΧΦεΤϯδχΞϦϯάͷྲྀߦ • ϨδϦΤϯεΤϯδχΞϦϯάͱ Safety-II • SRE
ͱΧΦεΤϯδχΞϦϯάͷؔ • ΞϯνϑϥδϟΠϧͳ Web γεςϜཱ͕͢Δ͔ Web System Artchitecture ݚڀձ #4 (@itkq) 3 -
ΧΦεΤϯδχΞϦϯάͷྲྀߦ • ʮΧΦεΤϯδχΞϦϯάʯ͕ॳΊͯొͨ͠ͷ 2015 1 • "Chaos Engineering" ϖʔύʔ 2016
2 • "Chaos Engineering" ຊ 2017 3 • Gremilin: Failure as a Service4 (2017ʙ) • ࣮ફͱ͔֓͘೦ͱͯ͠·͍ͬͯΔ 4 h$p:/ /principlesofchaos.org 3 h$ps:/ /www.oreilly.com/library/view/chaos-engineering/9781491988459/ 2 Chaos Engineering, Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Jus@n Reynolds, Casey Rosenthal 1 h$ps:/ /medium.com/ne2lix-techblog/chaos-engineering-upgraded-878d341f15fa Web System Artchitecture ݚڀձ #4 (@itkq) 5 -
ΧΦεΤϯδχΞϦϯάͱ ࢄγεςϜʹ͓͍ͯγεςϜ͕ෆ҆ఆͳঢ়ଶʹ͑Δ͜ͱͷग़ དྷΔڥΛߏங͢ΔͨΊͷݕূͷن — ΧΦεΤϯδχΞϦϯάͷݪଇ4 4 h$p:/ /principlesofchaos.org Web System
Artchitecture ݚڀձ #4 (@itkq) 6 -
ΧΦεΤϯδχΞϦϯάԿ͕৽͔ͬͨ͠ͷ͔ • πʔϧ? • ϥϯμϜʹյ͢͜ͱ? • ҙਤతʹյ͢͜ͱ? • ຊ൪Ͱյ͢͜ͱ? •
ܧଓతʹյ͢͜ͱ? Web System Artchitecture ݚڀձ #4 (@itkq) 7 -
ΧΦεΤϯδχΞϦϯάԿ͕৽͔ͬͨ͠ͷ͔ Amazon, Google, Microso2, Facebook ͳͲͷاۀͰɺࣗͷγ εςϜͷϨδϦΤϯεΛςετ͢ΔͨΊͷಉ༷ͳٕज़Λద༻ͯ͠ ͍ͨɻզʑͷۀքʹݱΕͨ͜ͷنΛܗ͢ΔΞΫςΟϏςΟΛ ʮΧΦεΤϯδχΞϦϯάʯͱݺͿ —
Ne%lix2 2 Chaos Engineering, Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Jus@n Reynolds, Casey Rosenthal Web System Artchitecture ݚڀձ #4 (@itkq) 8 -
ϨδϦΤϯεΤϯδχΞϦϯά5 • ࣾձɾٕज़γεςϜͷϨδϦΤϯτੑΛ্ͤ͞ΔͨΊͷํ๏ 6 • ϨδϦΤϯε: ྗੑɾ෮ݩྗɾճ෮ྗͷ༏Εͨঢ়ଶΛࢦ֓͢೦ 6 ϨδϦΤϯεΤϯδχΞϦϯά͕ࢦ҆͢શ Safety-II
ͱͦͷ࣮ݱ๏, ଜਖ਼, IEICE Fundamentals Review Vol.8 No.2 5 Resilience Engineering - Concepts and Precepts, Ashgate Publishing Ltd., E.Hollnagel, D.D.Woods, and N.Leveson, Eds., Aldershot, England, 2006 Web System Artchitecture ݚڀձ #4 (@itkq) 10 -
ࣾձɾٕज़γεςϜʹ͓͚Δͷଊ͑ํ 1. ࣌ؒෆมͰͳ͘ৗʹมԽ͢Δ 2. ෆશͳใΛͱʹॏཁͳҙࢥܾఆ͕͞ΕΔ 3. རӹޮΛٻ͢Δ͜ͱ͕ཁٻ͞ΕΔ͕ɺͦͷྗͷ݁Ռ҆ શͷ༨༟͕ΒΕΔ͜ͱଟ͘ɺةݥͳঢ়ଶυϦϑτ͢Δ ͕͋Δ 4.
҆શॏཁͰ͋Δ͕҆શͦͷͷ͕γεςϜಈ࡞ͷతͰͳ͍ Web System Artchitecture ݚڀձ #4 (@itkq) 11 -
ʮ҆શʯ֓೦ͷݶք • ैདྷͷʮ҆શʯͷྫ • ʮ·͘͠ͳ͍ࣄଶ͕ى͜Βͳ͍͜ͱʯ • ʮڐ༰Ͱ͖ͳ͍ϦεΫ͕ͳ͍͜ͱʯ • ΧλετϩϑΟʔʹଠଧͪͰ͖ͳ͍ •
ྫ͑౦ຊେࡂ Web System Artchitecture ݚڀձ #4 (@itkq) 12 -
ʮ҆શʯ֓೦ͷݶք • աͳӨڹྗ • ʮࣄނθϩ͕ݱࡏ n ܧଓதʯͷΑ͏ͳඪޠ • ࠜڌͷͳ͍ࣗݾա৴ •
ʮ͜Ε͚ͩపఈͯ҆͠શΛक͍ͬͯΔͷ͔ͩΒաࠅࣄނͳͲ ى͜Δ͕ͣͳ͍ʯ Web System Artchitecture ݚڀձ #4 (@itkq) 13 -
Safety-I ͔Β Safety-II • Safety-I • ैདྷͷ൱ఆܗ͔ͭ੩తʹఆٛ͞Εͨʮ҆શʯ • Safety-II10
• ʮγεςϜ͕େ֎ཚͳͲʹΑͬͯ௨ৗ࣌ͷಈ࡞ঢ়ଶΛҡ࣋Ͱ͖ͳ͍߹ɺੑೳԼͤͯ͞ಈ࡞Ͱ͖Δʯ • ʮঢ়گ͕ճ෮ͨ͠Β͔ʹݩͷঢ়ଶ·ͨͦΕʹ४͡Δঢ়ଶʹ෮چͰ͖Δʯ • ϨδϦΤϯτੑͷ͋Δڍಈ͕Ͱ͖Δ͜ͱ • Safety-I Λશ໘൱ఆ͍ͯ͠ΔͷͰͳ͘ɺͦͷઌʹ͋Δͷ 10 E.Hollnagel, Safety-I and Safety-II - The Past and Future of Safety Management, Ashgate Publishing Ltd., Surrey, England, 2014. Web System Artchitecture ݚڀձ #4 (@itkq) 14 -
جૅͱͳΔ4ͭͷओཁͳೳྗ • ରॲͰ͖Δ • ྟػԠมͳରॲ·ͰΛؚΉ • ࢹͰ͖Δ • ϓϩΞΫςΟϒʹରԠ͢ΔೳྗΛ࣋ͭ͜ͱ͕·͍͠ •
༧ݟͰ͖Δ • ඞͣ͠σʔλۦಈͱݶΒͳ͍ • ֶशͰ͖Δ • ্هͷೳྗΛઈ͑ؒͳ্ͤ͘͞Δ͜ͱ Web System Artchitecture ݚڀձ #4 (@itkq) 15 -
҆શͷࢿײͷࠩҟ • Safety-I ϕʔε • Կى͜Βͳ͍͜ͱ͕·͍͠ͱ͍͏҉ͷԾఆʹΑΓɺ ࢿߦಈֻ͚ࣺͯอݥͷΑ͏ʹଊ͑ΒΕ߅͕ੜ·Ε͍͢ • Safety-II ϕʔε
• తಈ࡞ͷܧଓͰ͋ΔͨΊɺͦͷՄೳੑΛߴΊΔࢿԾ ʹେ֎ཚڥͷมԽ͕ى͜Βͣͱਖ਼ੑΛओுͰ͖Δ Web System Artchitecture ݚڀձ #4 (@itkq) 16 -
Safety-II ͷධՁͷ͠͞ • ҆શͷܧଓྗͱՌධՁʹؔ͢ΔδϨϯϚ • ରͷγεςϜ͢Ͱʹ͋Δఔߴ͍҆શੑΛୡࡁΈͰ͋ ΔͨΊɺࣄނͳͲͷΞτΧϜධՁͰଌΔ͜ͱ͍͠ • ϓϩηεධՁͰͷิ͕ඞཁ Web
System Artchitecture ݚڀձ #4 (@itkq) 17 -
Web γεςϜͰͲ͏͔? • ৗʹมԽ͢Δ • ۜͷؙͳ͍ • ӡ༻Λݮ๏ͰධՁ͞ΕΔͷͭΒ͍ • ...
• ʮ҆શʯ=>ʮ৴པੑʯɺʮࣄނʯ=>ʮোʯʹஔ͖͑ͯΈΔ ͱͲ͏͔ Web System Artchitecture ݚڀձ #4 (@itkq) 18 -
Web γεςϜͷ৴པੑ • ଟ͘ͷ Web γεςϜͰॏཁͳࢦඪ • ৴པੑ 100% Λඪʹ͢Δ͜ͱؒҧ͍7
• Web γεςϜͷ৴པੑΛ੍ޚ͢Δํ๏ => SRE • SRE ϨδϦΤϯεΤϯδχΞϦϯάͷ1ͭͷ࣮ͱ͍͑Δ 7 Site Reliability Engineering, Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy Web System Artchitecture ݚڀձ #4 (@itkq) 19 -
SLO ͱΤϥʔόδΣοτ • SLO (αʔϏεϨϕϧඪ) Λຬ͍ͨͯ͠ΔݶΓϦϦʔεՄೳ • => ΤϥʔόδΣοτ͕͍ͬͯΔ •
ΤϥʔόδΣοτ͕͍ͬͯͳ͍߹ • γεςϜͷϨδϦΤϯεΛߴΊΔ • SLO Λ؇ΊΔ Web System Artchitecture ݚڀձ #4 (@itkq) 20 -
SRE จ຺ͰͷϨδϦΤϯτੑͷධՁ • SLO తͳධՁج४ʹͳΒͳ͍ • ҰํͰ Web γεςϜͷোࣗવࡂʹൺΔͱ࠶ݱ͍͢͠ •
ো͕ى͜ΔͷΛ͚ͭͩͰͳ͘ɺোΛΤϛϡϨʔτ͢Δ ͜ͱͰγεςϜͷϨδϦΤϯτੑͷϓϩηεධՁ͕ߦ͑Δ • ͜Ε͕ΧΦεΤϯδχΞϦϯά Web System Artchitecture ݚڀձ #4 (@itkq) 21 -
ΧΦεΤϯδχΞϦϯά࠶ߟ • Web γεςϜͷ৴པੑ 100% Λඪʹ͢Δ͜ͱؒҧ͍ • SRE ͷత "αʔϏεͷ
SLO ΛԼճΔ͜ͱͳ͘มߋͷͷ࠷େԽΛ ٻ͢Δ͜ͱ7" • ΤϥʔόδΣοτΛ͍Ռͨ͞ͳ͍ͨΊʹ Safety-II ΛߴΊΔඞཁ͕͋Δ • ϨδϦΤϯτੑΛϓϩηεධՁ͢ΔͨΊͷํ๏ͱͯ͠োΛΤϛϡϨʔτ ͢ΔΧΦεΤϯδχΞϦϯά͕͋Δ 7 Site Reliability Engineering, Betsy Beyer, Chris Jones, Jennifer Petoff and Niall Richard Murphy Web System Artchitecture ݚڀձ #4 (@itkq) 22 -
ΧΦεΤϯδχΞϦϯάͰಘΒΕΔ͜ͱ • Known unknown ͳ੬ऑੑͷݕূ • ྫ: ΠϯελϯεΛಥવམͱͨ͠߹ʹ SLI ͕Ͳ͏มԽ͢Δ͔
• Unknown unknown ͳͷൃݟ • ྫ͑૬ؔ͢Δͷ • ྫ: ϨδϦΤϯεΛߴΊΔͨΊͷϑΥʔϧόοΫΩϟογϡ͕Ҿ͖ى͜͢ෆ߹11 11 h$ps:/ /medium.com/ne2lix-techblog/from-chaos-to-control-tes<ng-the-resiliency-of-ne2lixs-content-discovery- pla2orm-ce5566aef0a4 Web System Artchitecture ݚڀձ #4 (@itkq) 23 -
ΧΦεΤϯδχΞϦϯάͷݶք • Unknown unknown ͳো੬ऑੑΤϛϡϨʔτͰ͖ͳ͍ • ྫ͑છతͳͷ8 • ྫ: Linux
Leap Second bug • "ଟ༷ੑ" ཱ͔ͪ͏खஈͷ1ͭ8 • ଟ༷ੑ...? 8 h$ps:/ /www.gremlin.com/blog/adrian-cockro9-chaos-engineering-what-it-is-and-where-its-going-chaos- conf-2018/ Web System Artchitecture ݚڀձ #4 (@itkq) 24 -
੬͞ (ΞϯνϑϥδϟΠϧੑ) • ੬͞ͷରͱ͞ΕΔ֓೦9 • ྫ • ے: ෛՙΛ͔͚Δ͜ͱͰҎલΑΓڧ͘ͳΔ •
ใ: ΊΔΑΓյ͢ྗͷ΄͏͕ྐʹͳΔ 9 ੬ऑੑʦ্ʧ――ෆ࣮֬ͳੈքΛੜ͖ԆͼΔ།Ұͷߟ͑ํ, φγʔϜɾχίϥεɾλϨϒ (ஶ), ݄ Ӵ (༁), ઍ ༿ හੜ (༁) Web System Artchitecture ݚڀձ #4 (@itkq) 26 -
੬͞ (ΞϯνϑϥδϟΠϧੑ) • ੬͞: িܸͰऑ͘ͳΔੑ࣭ • ϩόετੑ: িܸʹରͯ͠มԽ͠ͳ͍ੑ࣭ • ϨδϦΤϯτੑ:
িܸʹରͯ͠దԠ͢Δੑ࣭ • ੬͞: িܸͰڧ͘ͳΔੑ࣭ Web System Artchitecture ݚڀձ #4 (@itkq) 27 -
੬͍γεςϜͷ֊ߏ • ੬͞ͷཪଆʹ֊ߏ͕͋Δ9 • Web γεςϜશମ͕੬͘ʮਐԽʯ͢ΔͨΊʹݸʑͷ Web γεςϜ͕੬͘ɺഁ͢ΔՄ ೳੑΛ͍࣋ͬͯΔ͜ͱ͕͔ܽͤͳ͍ •
Web γεςϜ͕੬͍ͨΊʹݸʑͷίϯϙʔωϯτ͕੬͘ɺഁ͢ΔՄೳੑΛ͍࣋ͬͯΔ ͜ͱ͕͔ܽͤͳ͍ • ίϯϙʔωϯτ͕੬͍ͨΊʹݸʑͷϗετ͕੬͘ɺഁ͢ΔՄೳੑΛ͍࣋ͬͯΔ͜ͱ͕ ͔ܽͤͳ͍ => ϗετͷଟ༷ੑ 9 ੬ऑੑʦ্ʧ――ෆ࣮֬ͳੈքΛੜ͖ԆͼΔ།Ұͷߟ͑ํ, φγʔϜɾχίϥεɾλϨϒ (ஶ), ݄ Ӵ (༁), ઍ ༿ හੜ (༁) Web System Artchitecture ݚڀձ #4 (@itkq) 28 -
·ͱΊ • ϨδϦΤϯεΤϯδχΞϦϯάʹ͍ͭͯಋೖͨ͠ • ϨδϦΤϯεΤϯδχΞϦϯάɾSREɾΧΦεΤϯδχΞϦϯ άͷؔΛઆ໌ͨ͠ • ੬͍ Web γεςϜͷߏΛઆ໌ͨ͠
Web System Artchitecture ݚڀձ #4 (@itkq) 29