Before Chaos Engineering

golang.tokyo#24
https://golangtokyo.connpass.com/event/129067/

Transcript

1. ©2019 Wantedly, Inc. Chaos Engineeringͷ खલͷ࿩ golang.tokyo #24 May 20, 2019 - Hazumi Ichijo (rerost/hazumirr) ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰͰ͖Δ͜ͱ
2. ©2018 Wantedly, Inc. ࣗݾ঺հ Ұᑍ
   ୺੅
   !IB[VNJSS!SFSPTU
   4FSWFS
   4JEF
   &OHJOFFS
   JO
   8BOUFEMZ
   (P
   3BJMT
   3FBDUʜ
   ग़਎
   ๺ւಓ
3. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔
4. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ => ೺Ѳ͍ͯ͠Δͭ΋Γ͕ͩͦΕ͸ຊ౰͔ʁ
5. ©2019 Wantedly, Inc.
6. ©2019 Wantedly, Inc. ͦ͏ͩChaos Engineeringͩ
7. ©2019 Wantedly, Inc. ͦ΋ͦ΋Chaos Engineeringͱ͸ʁ
8. ©2019 Wantedly, Inc. > ΧΦεΤϯδχΞϦϯά͸ɺ෼ࢄγεςϜʹ͓͍ͯγεςϜ͕ෆ҆ ఆͳঢ়ଶʹ଱͑Δ͜ͱͷग़དྷΔ؀ڥΛߏங͢ΔͨΊͷݕূͷن཯ https://principlesofchaos.org/?lang=JAcontent#
9. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc… ෆ҆ఆͳঢ়ଶͱ͸
10. ©2019 Wantedly, Inc. 1. αʔϏε͕ෆ҆ఆͳ࣌ɺͲ͏͍ͬͨӨڹ͕ͰΔ͔ 2. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ɺͲΜͳӨڹ͕ͰΔ͔ ஌Γ͍ͨ͜ͱ
11. ©2019 Wantedly, Inc. αʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $
12. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌
13. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌
14. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # 4FSWJDF
    $ αʔϏε͕ෆ҆ఆͳ࣌
15. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε
16. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε
17. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε
18. ©2019 Wantedly, Inc. ґଘ͢ΔαʔϏε͕ෆ҆ఆͳ࣌ ࣗ෼ͷαʔϏε ଞͷαʔϏε
19. ©2019 Wantedly, Inc. Ͳ͏͍ͬͨ͜ͱΛ͢Δ΂͖͔ʁ
20. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# Ͳ͏͍ͬͨ͜ͱΛߦ͏͔
21. ©2019 Wantedly, Inc. 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ 3. ຊ൪؀ڥͰݕূΛ࣮ߦ͢Δ 4. ܧଓతʹ࣮ߦ͢ΔݕূͷࣗಈԽ 5. ӨڹൣғΛہॴԽ͢Δ https://principlesofchaos.org/?lang=JAcontent# 43&ͱͷڠྗ΍૊৫త ͳରԠ͕ඞཁ Ͳ͏͍ͬͨ͜ͱΛߦ͏͔
22. ©2019 Wantedly, Inc. 1. 1ඵؒͷಈը࠶ੜ਺ͳͲ 2. APIͷϨεϙϯε଎౓ͳͲ 1. ఆৗঢ়ଶʹ͓͚ΔৼΔ෣͍ͷԾઆΛཱͯΔ => ͦΕΛकΔͨΊͷ΋ͷ͕ॏཁɻͦͷ֬ೝ͕Ͱ͖Ε͹͋Δఔ౓҆৺ ྫ: AWS͕region͝ͱμ΢ϯͨ͠ࡍɺଞͷregionʹfail over͢ΔͳͲ ྫ
23. ©2019 Wantedly, Inc. 1. Πϯελϯε͕μ΢ϯ͢Δ 2. DB͕٧·ΔɾΞΫηεͰ͖ͳ͘ͳΔɹ 3. ґଘ͢ΔଞͷαʔϏε͕ෆ҆ఆʹͳΔ etc… 2. ࣮ੈքͷࣄ৅͸ଟ༷Ͱ͋Δ => ͜͏͍ͬͨཁૉΛೖΕΒΕΔΑ͏ʹ͢Δ = Fault Injection
24. ©2019 Wantedly, Inc. ͜ͷ2͚ͭͩͳΒQA؀ڥͰ ΞϓϦέʔγϣϯΤϯδχΞ͚ͩͰՄೳ
25. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ
26. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ ֬ೝ༻ͷίʔυΛೖΕΔͷ͕໘౗
27. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ
28. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ
29. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # { "interval": 1, "slow_response_option": { "enable": true, "time": 5 }, "server_error_option": { "enable": true } } Fault Injectionͱͦͷ֬ೝ
30. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε  ൪୆Λฦ͢
     ஗͘ͳΔ
     ઃఆ͍ͯ͠ΔλΠϜΞ΢ τΑΓϨεϙϯε͕஗͍
    JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ
31. ©2019 Wantedly, Inc. ࣗ෼ͷαʔϏε JOUFSDFQUPS 4FSWJDF
    " 4FSWJDF
    # Fault Injectionͱͦͷ֬ೝ
32. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ
33. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ
34. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC  ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ 1VCMJTI 4VCTDSJCF Fault Injectionͱͦͷ֬ೝ
35. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC  ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ 1VCMJTI 4VCTDSJCF ϥΠϒϥϦ͕ߦ͏ͷͰ JOUFSDFQUPSΛڬΊͳ͍
    Fault Injectionͱͦͷ֬ೝ
36. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ Fault Injectionͱͦͷ֬ೝ
37. ©2019 Wantedly, Inc. 4FSWJDF
    " 4FSWJDF
    # $MPVE
    1VC4VC 1VCMJTI 4VCTDSJCF 1SPYZ 1SPYZ  ॏෳ഑৴Λ͢Δ
     1VCMJTIͨ͠ॱ൪Ͱ഑৴͠ͳ͍ Fault Injectionͱͦͷ֬ೝ
38. ©2019 Wantedly, Inc. https://github.com/rerost/unstable https://github.com/rerost/chaos-pubsub
39. ©2019 Wantedly, Inc.
40. ©2019 Wantedly, Inc. σϝϦοτ  ݱঢ়ͷ֬ೝ͸Ͱ͖Δ͕͍ͭյΕΔ͔Θ͔Βͳ͍ɹ
     ςετ͢Δͷ͕ਓखͳͷͰ֬ೝ͕໘౗
     ςετ͕໢ཏͰ͖͍ͯͳ͍Մೳੑ͕͋Δ
41. ©2019 Wantedly, Inc. ·ͱΊ  खಈͰ΋͋Δఔ౓ͷݕূ͸Ͱ͖Δ
     ςετ༻ͷίʔυ͸ൈ͖΍͍͢ܗʹ͓ͯ͘͠ɹ
     2"؀ڥ͚ͩͰ΋ࢼ͓ͯ͘͠ҙຯ͸͋Δɹ
     σϝϦοτ͸ଟ͍ͷͰҰൠʹ͓͢͢ΊͰ͖Δ ํ๏Ͱ͸ͳ͍