Upgrade to Pro — share decks privately, control downloads, hide ads and more …

What is suitable for Chaos Engineering? ~ChaosC...

hata
November 18, 2019

What is suitable for Chaos Engineering? ~ChaosConf 2019 recap~

Chaos Engineering → Chaos Monkey → 「本番環境でサーバを落とす」 → うちでは無理そう、、
というよくある誤解を解きたいと思って喋ったスライドです。 ChaosConf 2019 におけるセッションのスライドの紹介も交えながら説明しています。

AWS DevDay Tokyo 2019 の『Chaos Engineering ~入門と実例~』
https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf

から、スライドをいくつか引用しています。こちらは Netflix 社の Chaos Engineering の取り組みを時系列でまとめるかたちで Chaos Engineering とは何かを説明しているので、 Chaos Engineering 自体に馴染みのない方はまずはこちらの資料をどうぞ。

hata

November 18, 2019
Tweet

More Decks by hata

Other Decks in Technology

Transcript

  1. What is suitable for Chaos Engineering? ChaosConf2019 recap @AWS LOFT

    TOKYO Fumihiko Hata | Solutions Architect, AWS Japan | 2019.11.11
  2. Agenda Chaos Engineering  (* &) &)   ChaosConf

    2019.9'-% @SanFrancisco/ !"+$#,  What is suitable for Chaos Engineering?
  3. Agenda 1. Industries 2. Microservices or Monorith 3. Server-based or

    Server-less 4. Cloud or on-prem 5. Production or Non-production env What is suitable for Chaos Engineering?
  4.   Chaos Engineering      Chaos

    Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
  5.   Chaos Engineering      Chaos

    Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
  6.   Chaos Engineering      Chaos

    Engineering ~~ AWS DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
  7.  EC 7 • .40 , • $/( , #1

    2)0 '6 • +5  !" 3%-*&
  8. Uber EMBRACING CHAOS! Paul Osman, Under Armour Ana Medina, Gremlin

    • uDestroy: Uber  Chaos Experiment •  Uber  Gremlin  Ana  https://speakerdeck.com/chaosconf/ embracing-chaos
  9. 9513 • ChaosConf &+ Gremlin 4$8  Chaos Engineering .(

     %1 </,=-9513 • :6952; ( FinTech * 0 95!" "( • #'7) >
  10. :624? National Australia Bank • ChaosConf &+ Gremlin 5$9 

    Chaos Engineering .'  %2=0,>-:624 •  ;7:63< ' FinTech ) 1 :6 "#!#' • /*8( @ NAB deploys Chaos Monkey to kill servers 24/7 - Software – Cloud - iTnews 2014  https://www.itnews.com.au/news/nab-deploys-chaos-monkey-to-kill-servers-24-7-382285
  11. 9513> Capital One • ChaosConf %* Gremlin 4#8  Chaos

    Engineering -&  $1 </+=,9513 • :6952; & FinTech ( 0 95!" "& • .)7' ? https://twitter.com/CapitalOneTech/status/ 1194995848062623744 20191115
  12. NI@CQ TLPT (Threat-Led Penetration Test) H5$,%" *,(+ 2018710>NI8NI.M #,')9/2 0F

    =O<K4:NIAP6  TLPT ;L  -J *!,& *!,&BG 3?D1E !  TLPT  # "   https://www.fsa.go.jp/common/about/research/20180516.html
  13. NHBES TLPT (Threat-Led Penetration Test) G:(.#)'%,.!+-$#% M86G:(.#)'%,. !+-$#%QTLPTRO 74A 2IL

    TLPT  Chaos Experiment ?81 =>@CD  0 PF 5 9 /;JK NH BE -"&3< ,*.% !  TLPT  # "   https://www.fsa.go.jp/common/about/research/20180516.html
  14. ^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0R<a\  '/ED $"#/7T

    !*+-/!,.  `@3^XK_`@HU>B Y]^XK_ '/ ED>BC8[1 >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT O<0R<a\ E D7T!*+-/!,.  '/ ED2= >4ED>B V6b ML$"% ?$"%<K Q9 5 GSP:<F ” 2.2.3. TLPT NA  2018131     TLPT   https://www.fsa.go.jp/common/about/research/20180516/TLPT.pdf
  15. ^XIRc TLPT (Threat-Led Penetration Test) “W;(/")&%-/!,.$"% O<0R<a\  '/ED $"#/7T

    !*+-/!,.  `@3^XK_`@HU>B Y]^XK_ '/ ED>BC8[1 >BV6b  ML$"%” 2.2.1. $"%JZ  “TLPT O<0R<a\ E D7T!*+-/!,.  '/ ED2= >4ED>B V6b ML$"% ?$"%<K Q9 5 GSP:<F ” 2.2.3. TLPT NA  20181 31     TLPT   https://www.fsa.go.jp/common/about/research/20180516/TLPT.pdf Chaos Engineering    Gameday    
  16. Microservices )/ 1'" • FH N@< 1'"T) / 1'"U (.1!-0B>

    1%$,4IK; !"%*SG(/1$ • 6 1'" API R 5 •   1'"<OC M= 9JFH $1*?A • ;E N@ 2 DLV  1'"T(/#"U3 :J7 (.1!-0QP+&." 8
  17. Microservices • 7T( .U"1&$Q Failure BS9R  Chaos Experiment @HPF"1

    &$;5A%1) J><? • Netflix  Amazon ( ."1&$ DK I= • 2G Netflix "1&$Q  ',!1#+/0-*  Failure Context 4E3L O"1&$Q 8 6M:CN 
  18. Monorith THINK BIG: CHAOS TESTING A MONOLITH https://speakerdeck.com/chaosconf/think-big-chaos-testing-a-monolith • 3;8JC$;-&/6(*.

    ;2 Mailchimp ZD  Mailchimp RMI  SRE Caroline '(%5: • 2000<TX4,7& PHP /7 ";%5: Chaos Engineering GY H>  • 1!9$;-&%&)2 OB[FE  A#:0;+ :* Experiment GP   @?LQK G=N VSUW
  19. Server-based or Server-less Server-based • )!&*($938 Failure  Injection /;

    • shutdown now • tc qdisc add dev eth0 root netem delay 2000ms •  "'+%)*($  Injection 26/; Serverless • )!&*($938 4. 5>,? •  )!&($931 =0<  + +( #' • 078"'+%)($  Injection -:0<
  20. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. | | | | | | | | | | | | | | | 56 Microservices & % 7-2+ • 2014.101 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) #(/,)0  '$("'!(' 3 !' Failure *4 https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2
  21. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. | | | | | | | | | | | | | | | 9: Microservices ("' <06. • 20142105 FIT: Failure Injection Testing (Netflix TechBlog) • Request Context (Failure Context, FIT Context) %+3/,4   *&+$*#+!* 7 #* Failure -8  https://medium.com/netflix-techblog/fit-failure-injection-testing-35d8e2a9bb2 a little too wild 1)' ;
  22. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. /  FIT ,%28 Chaos Monkey, Simian Army ! EC2 Instance  API terminate  tc NW 7' 6"  3+2* Blast Radius: -1#) 5; • 04$.%9%  • (:4$& 
  23. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. S FIT NBV_9 Chaos Monkey, Simian Army ; EC2 Instance  API  terminate  tc  NW ^E \> "5,264"/ WJVI Blast Radius: PU?G [d • :TY@ QB9aB  • FcY@ D   FIT ; UA #2"!5+ XL bCO= *(+ ,2$07&]<  DB !#)( ^E  !-3%7'1564"/ HA 8K !-3%7'1 564"/HA Z`  • R."5+ • MaB Y@
  24. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. OSS Tools: Serverless aws-lambda-chaos-injection Chaos Injection library for AWS Lambda Python I?< α .6$04J 2/$*2!6,6 Adrian Hornsby  AWS  Principal Evangelist Adrian  Gremlin  Ana &(#04  Breaking Containers: Chaos Engineering for Modern Applications on AWS (CON310)  ;@  %1 + BF   https://medium.com/@adhorn/failure-injection-gain-confidence-in-your-serverless-application-ce6c0060f586 7D latency injection )"36' Lambda function -4+19 >C 555 ↑3%/4%8G= ↓ Duration >H Delay : AE 
  25. Cloud or on-prem • )$'&=5  Chaos Experiment ,-D9 

     •  Netflix )$' AWS @C G Chaos Monkey F?*/>7  #( EA.F  Chaos Engineering 6 +")%)! H  3<;01B2:84  I
  26. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. | | | | | | | | | | | | | | | Netflix  AWS -. "+( • 2010$12'14& Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) ,/  Amazon   0%4)* • 2010$12'16& 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS !# 5 https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c
  27. © 2019, Amazon Web Services, Inc. or its affiliates. All

    rights reserved. | | | | | | | | | | | | | | | Netflix  AWS FK 04D@ • 2010812>14< Four Reasons We Choose Amazon’s Cloud as Our Computing Platform (Netflix TechBlog) EM.'+/#.),"$(/*  Amazon ,%N:4 AB • 2010812>16< 5 Lessons We’ve Learned Using AWS (Netflix TechBlog) AWS 2 6 5 -" . https://medium.com/netflix-techblog/four-reasons-we-choose-amazons-cloud-as-our-computing-platform-4aceb692afec https://medium.com/netflix-techblog/5-lessons-weve-learned-using-aws-1f2a28588e4c P7 (failure) O=J;? HICP7L (fail)   Chaos Monkey 5  /& ,.!*93G1
  28. AWS  Chaos Experiment  API Chaos Engineering ~~ AWS

    DevDay 2019 https://pages.awscloud.com/rs/112-TZM-766/images/E-2.pdf
  29. GameDay game-day: 8F1= • !(")G  !(")&+!   

    !(")G >%+59  I • ?B2@A3C6-:59ED • AWS GameDay: <.+%0AWS ,;/  4HA3!(")&+!  *$ '*87#*! https://eow.alc.co.jp/search?q=game-day https://aws.amazon.com/gameday/
  30. ! • Chaos Engineering UH#]PQXZ% S>"e 3-/25+.*-W_#8;"aB • cIK bc&

    j gA`h C<&:R''#%# (#^Y ,-.4TJ?[L$D M)9 ENF • fG& 1706 iOdP Dev V@# GameDay C<#=%\ $!