CloudWatch複合アラームでELBの5XXをいい感じに検知しようとしたらうまくいかなかった話 / cloudwatch alarm elb 5xx

2023/05/29開催「JAWS-UG SRE支部 #6」での発表資料です。

Transcript

1. 
   LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
   ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
   ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(
   43&ࢧ෦
   
2. ࣗݾ঺հ LHPUP w ςοΫϦʔυɾϥʔϝϯ޷͖
   w "84
   $PNNVOJUZ
   #VJMEFS %FW5PPMT
   w าͷςοΫ ٕज़ϒϩά
   w ࣗ࡞"84ؔ࿈πʔϧͷ044։ൃ
   w 5XJUUFS
   !@TUFQ@UFDI
   ‣ LHPUP าͷςοΫ
3. $MPVE8BUDIෳ߹ΞϥʔϜͰ
4. &-#ͷ99Λ
5. ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
6. ͏·͍͔͘ͳ͔ͬͨ࿩
7. ͍͍ײ͡ʁ w &-# "-# ͷΤϥʔ௨஌ $MPVE8BUDI
   "MBSN
   ‣ )551$PEF@&-#@ccc@$PVOU
    εςʔλε͝ͱʹ௨஌͍ͨ͠
   ‣ )551$PEF@&-#@99@$PVOU
    cccͷ৔߹͸99௨஌͠ͳ͍
   㾎ॏෳ௨஌Λഉআ͍ͨ͠ ʮ99΋΋དྷͯ͠·͏ʯͷΛආ͚Δ
    cccҎ֎ͷΤϥʔ͚ͩ99Ͱ௨஌͢Δ
   㾎
   
   ͳͲͷͱ͖͚ͩ௨஌͢Δ
   w ͦΜͳ΋ͷΛʰ$MPVE8BUDIෳ߹ΞϥʔϜʱͰ࡞ͬͯΈͨ
8. $MPVE8BUDIෳ߹ΞϥʔϜͱ͸ʁ
9. $MPVE8BUDIෳ߹ΞϥʔϜ w ଞͷෳ਺ͷΞϥʔϜͷঢ়ଶΛ΋ͱʹ
   w ࣜͰಈతʹ൑அͰ͖ΔΞϥʔϜ (ALARM("CPUUtilizationTooHigh") OR ALARM("DiskReadOpsTooHigh")) AND OK("NetworkOutTooHigh") $166UJMJ[BUJPO5PP)JHI͕ΞϥʔϜঢ়ଶ
   ·ͨ͸
   %JTL3FBE0QT5PP)JHI͕ΞϥʔϜঢ়ଶ "/%
   /FUXPSL0VU5PP)JHI͕ਖ਼ৗঢ়ଶ ΞϥʔϜঢ়ଶ
10. ΍ͬͯΈͨ
11. ΍ͬͯΈͨ ෳ߹ΞϥʔϜ 99୅ΘΓͳͲͰൃՐͤ͞Δ༻
12. ΍ͬͯΈͨ YY͕ΞϥʔϜঢ়ଶ "/%
    ͕ΞϥʔϜঢ়ଶ
    ·ͨ͸
    ͕ΞϥʔϜঢ়ଶ /05 ˞ ͕ͳ͍ͷ͸
    ݕূ্ͷཧ༝Ͱ͢ ͩͱΞϥʔϜ
    ͩͱਖ਼ৗ
13. ΍ͬͯΈͨ ΞϥʔϜ ͳͲ΋ಉ༷ʹ࡞੒
14. ΍ͬͯΈͨ 99ΞϥʔϜ ࣮ࡍͷൃՐ͸ෳ߹ΞϥʔϜͷํͰߦ͏ͷͰ
    ͪ͜Β͸ΞΫγϣϯແޮʹ
    ͨͩͷΞϥʔϜࡐྉͱͯ͠࡞੒
15. ΤϥʔΛग़ͯ͠Έͨ
16. ௨஌͕དྷͨʂ
17. ෳ߹ΞϥʔϜͷํ͕དྷͨʂ
18. ͭ·Γ
    ʮ   Ͱͳ͍99ʯ
    ͷํ
19. ͭ·Γɺͱ͔ʂ
20. Ͱ΋Ͱ΋Ͱ΋Ͱ΋
    ͳ͍εςʔλεͳΜͯ
    ͦΜͳ؆୯ʹى͜Δ͔ɾɾɾʁ
21. ͦͷ͞Βʹ਺ඵޙ
    ΋͏݅௨஌͕ɾɾɾ
22. 
23. ঢ়گΛ੔ཧͯ͠Έͨ w ෳ߹ΞϥʔϜ͕དྷͨ
    ‣ 
    
    
    Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ
    ‣ 
    
    
    ͷͱ͖͸ɺ͜ͷΞϥʔϜ͸ൃՐ͠ͳ͍
     ͔֬ʹͦͷ࣌͸͸ൃՐͤͣ99ͷΈൃՐ͍ͯͨ͠͸ͣ
    ‣ ͭ·Γͱ͔͕ൃੜͨ͠ʁ
    w ͦͷ਺ඵޙʹɺΤϥʔ͕དྷͨ
    ‣ Ͱ΋͖ͬ͞ෳ߹ΞϥʔϜ͕དྷͨͬͯ͜ͱ͸ɺͰ͸ͳ͍͸ͣɾɾɾ
24. ঢ়گΛ੔ཧͯ͠Έͨ w $MPVE8BUDIϝτϦΫεΛݟͯΈͨ
    ‣ )551$PEF@&-#@99@$PVOU
     ݅
    ‣ )551$PEF@&-#@@$PVOU
     ݅ ͳͲͷɺͦͷଞͷΤϥʔ͸
    ى͖͍ͯ·ͤΜͰͨ͠ɻ
25. ঢ়گΛ੔ཧͯ͠Έͨ w Τϥʔ͕
    ݅
    ൃੜʢҙਤతʹʣ
    w 99୅ΘΓͷෳ߹ΞϥʔϜ͕݅௨஌
    ‣ 
    
    
    Ͱͳ͍99ͷͱ͖ʹ௨஌ ͢Δ͸ͣ
    w ਺ඵޙ )551$PEF@&-#@͕݅௨஌ ݅ͷΤϥʔʹରͯ͠
    ෳ߹ΞϥʔϜ΋ΞϥʔϜ΋
    ྆ํ௨஌͕དྷͨ ॏෳ௨஌͞Εͯ͠·ͬͨ
26. ͏·͍͔͘ͳ͔ͬͨ
27. Α͘ݟͯΈͨ
28. Α͘ݟͯΈͨ
29. Α͘ݟͯΈͨ
30. Α͘ݟͯΈͨ w )551$PEF@&-#@99@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ
    w )551$PEF@&-#@@$PVOU
    ‣ ൃੜ ঢ়ଶͷߋ৽ ࣌ࠁɿ 99͕ઌʹൃՐ͠ɺͦͷޙ͕ൃՐͨ͠ ෳ߹ΞϥʔϜͷλΠϛϯάͰ͸
    ͔֬ʹ99͔͠ൃՐ͍ͯ͠ͳ͍
    ͸ൃՐ͍ͯ͠ͳ͍ ࣜ௨Γͷڍಈʹͳ͍ͬͯΔ
31. 99͕ઌʹൃՐͯ͠
    ͦͷޙ۩৅εςʔλε͕
    ൃՐ͢Δͷ͔ɾɾɾʁ
32. ೦ͷͨΊ؀ڥΛ࡞Γ௚ͯ͠
    ࠶౓࣮ߦͯ͠Έͨ
33. ࠓ౓͸
    99ΑΓઌʹ͕ൃՐͨ͠
34. ൃՐλΠϛϯά ඵ਺ ͸
    ຖճಉͩͬͨ͡
35. ൃՐධՁλΠϛϯά͸
    ΞϥʔϜΛ࡞੒ͨ͠
    λΠϛϯά ࣌ࠁ ʹΑΔʁ
36. ඵ਺͸ҧ͏ɾɾɾ
38. ԿʹͤΑ
    ධՁλΠϛϯά͸ຖ෼ಉ͡ඵ਺
39. ΞϥʔϜͷධՁλΠϛϯάͷલޙͲͪΒͰ Τϥʔ͕ൃੜ͢Δ͔࣍ୈͰॱ൪͕มΘΔ T T T T 99ධՁ ධՁ ͜͜ͰΤϥʔ͕ى͖ͨΒ͕ઌʹൃՐ ͜͜ͰΤϥʔ͕ى͖ͨΒ99͕ઌʹൃՐ
40. ΤϥʔͷൃੜλΠϛϯάͰ
    ΞϥʔϜൃՐॱ൪͕มΘΔͷ͸ࠔΔ
41. ͜ΕͰऴΘΓ͔ʁ
42. ͍΍
43. ·ͩ΍ΕΔ
44. αϓϨοαʔΞϥʔϜ
45. αϓϨοαʔΞϥʔϜ IUUQTEPDTBXTBNB[PODPNKB@KQ"NB[PO$MPVE8BUDIMBUFTUNPOJUPSJOH$SFBUF@$PNQPTJUF@"MBSNIUNM
46. αϓϨοαʔΞϥʔϜ ࣮ߦ͢·͢
47. αϓϨοαʔΞϥʔϜ w બ୒ͨ͠ΞϥʔϜ͕ΞϥʔϜঢ়ଶʹͳΔͱෳ߹ΞϥʔϜͷΞΫγϣϯ͕ແޮʹͳΔ
    ‣ αϓϨοαʔΞϥʔϜͷൃՐΛࢦఆඵ਺଴ͪɺෳ߹ΞϥʔϜͷൃՐΛ஗Ԇɾ཈੍
    w αϓϨοαʔΞϥʔϜ͸͔ͭ͠બ΂ͳ͍
    ‣ 
    
    
    Λશ෦બ͹ͳ͍ͱࠓճͷ໨త͸ୡ੒Ͱ͖ͳ͍
     ͜ΕΒ͕ൃՐ͢Δ͔Ͳ͏͔଴͔ͬͯΒෳ߹ΞϥʔϜΛධՁ͍ͤͨ͞
     ΋࣌ؒࠩ͠ͰൃՐ͢ΔͳΒෳ߹ΞϥʔϜͷൃՐΛࢭΊ͍ͨ
    ‣ 
    
    
    ͷෳ߹ΞϥʔϜΛ࡞੒ͯ͠ɺͦΕΛબ୒͢Ε͹ྑ͍
     UFTUHPUPFMCTVQQSFTTJPO
    ෳ߹ΞϥʔϜ
    ɹɹ
    l"-"3. UFTUHPUPFMC
    03
    "-"3. lUFTUHPUPFMC
    03
    ʜz
48. αϓϨοαʔΞϥʔϜ
49. ΍ͬͯΈͨ
50. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ UFTUHPUPFMC99 UFTUHPUPFMC UFTUHPUPFMCTVQQSFTTJPO ࠷ॳͷݕূ࣌ͱಉ͘͡ΤϥʔΛൃੜͤͨ͞Β
    99͕ઌʹൃՐͨ͠
    ͱෳ߹ͷॏෳൃՐࣦͨ͠ഊύλʔϯ௨Γ
51. ͢Δͱɾɾɾ
52. ෳ߹ΞϥʔϜ ࣮࣭99 ͸
    ൃՐ ௨஌ ͤͣ
53. ͚ͩ௨஌͕དྷͨʂʂʂ
54. ॏෳഉআ੒ޭʂ
55. ΍ͬͯΈͨɿαϓϨοαʔΞϥʔϜ 99ൃՐʹΑΓҰ࣌తʹΞϥʔϜঢ়ଶʹ͸ͳ͍ͬͯΔ͕
    ෼ؒ 8BJU1FSJPE ͸ΞΫγϣϯ͕཈੍͞ΕΔ
    ͦͷޙൃՐʹΑͬͯ0,ঢ়ଶʹ໭ͬͨ
    ʢ͕ൃՐ͠ͳ͚Ε͹ΞΫγϣϯ͕ى͜Δ͸ͣʁʣ
56. 
    LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(
    43&ࢧ෦
    
57. 
    LHPUP าͷςοΫ $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ
    ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β
    ͏·͍͔͘ͳ͔ͬͨ࿩ +"846(
    43&ࢧ෦
    
58. ஫ҙ w ࣮ࡍʹͳͲͷΤϥʔ    Ҏ֎ Λൃੜͤ͞ΔݕূΛ͍ͯ͠ ͳ͍ͷͰɺͩͱͪΌΜͱෳ߹ΞϥʔϜ͕ൃՐ͢Δ͔Ͳ͏͔ͷ֬ೝ͸ͯ͠ ͍·ͤΜ
    ‣ μϝͩͬͨΒ͍͢·ͤΜɾɾɾ
59. ஫ҙ w ࠓճͷख๏Ͱ͸ɺ͏·͍ͬͨ͘ʢ૝ఆ௨Γͷڍಈʹͳͬͨʣͱͯ͠΋ɺ    ͱͦΕҎ֎ͷ99Τϥʔ͕ಉ࣌ʢಉධՁظؒ಺ʣʹൃੜͨ͠৔߹ɺ ෳ߹ΞϥʔϜ ࣮࣭99ϝτϦΫε ͷํͷ௨஌͕དྷ·ͤΜɻ
    ‣ ྫ͑͹ͱ͕ಉ࣌ʹى͖ͨͱ͖ɺʢ۩৅ʣͷ௨஌͚ͩདྷΔ
    w ͦΕ͸࣮ࡍʹΤϥʔ͕ى͖ͨ࣌͸݁ہ"84ίϯιʔϧͷάϥϑΛݟʹߦ͘ͱ ࢥ͏ͷͰͦΕͰ൑ผ͢Ε͹͍͍͔ͳͱ͍͏ํ޲ͰɺؤுΒͳ͍͜ͱʹ͠·͠ ͨɻ
60. ࢀߟɿϒϩάهࣄ าͷςοΫ w $MPVE8BUDIෳ߹ΞϥʔϜͰ&-#ͷ99Λ͍͍ײ͡ʹݕ஌͠Α͏ͱͨ͠Β ͏·͍͔͘ͳ͔ͬͨ࿩
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZ
61. એ఻ɿࣗ࡞"84ؔ࿈πʔϧ044 w EFMTUBDL
    "84
    $MPVE'PSNBUJPOελοΫڧ੍࡟আπʔϧ
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZEFMTUBDL
    w DMT
    4όέοτߴ଎࡟আɾۭʹ͢Δπʔϧ όʔδϣχϯάରԠ
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZDMT
    w MBNWFS
    -BNCEBϥϯλΠϜόʔδϣϯݕࡧπʔϧ Ϧʔδϣϯԣஅ
    ‣ IUUQTHPUPLIBUFOBCMPHDPNFOUSZMBNWFS
62. એ఻ɿొஃ w  ໦
    d
    "84
    %FW
    %BZ
    
    5PLZP
    ೥࿈ଓ
    ‣ ʰ"84
    $%,ͰֶͿ(P'σβΠϯύλʔϯʙ*B$ʹ΋ίʔυઃܭʙʱ
63. 5IBOL
    :PV LHPUP าͷςοΫ