Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
LT_2023-01-26 不要になったAWSリソースを 削除してサービス障害起こした話
Search
tomoki inoue
January 26, 2023
Programming
0
270
LT_2023-01-26 不要になったAWSリソースを 削除してサービス障害起こした話
不要になったAWSリソースを 削除してサービス障害起こした話
tomoki inoue
January 26, 2023
Tweet
Share
Other Decks in Programming
See All in Programming
try! Swift Tokyo 2024 参加報告 / try! Swift Tokyo 2024 Report
hironytic
0
200
Random\Randomizer クラスで日常のあれこれを解決しよう! / Random\Randomizer class solves familiar trouble
cocoeyes02
0
210
Compose-View Interop in Practice (mDevCamp 2024)
stewemetal
0
120
二郎系ラーメンのコールで学ぶ AST 解析
memory1994
PRO
7
1.7k
PHPはいつから死んでいるかの調査
chiroruxx
1
380
SIMD Parallel Programming with the Vector API
josepaumard
0
120
StoreKit2によるiOSのアプリ内課金のリニューアル
kangnux
0
110
Anthropic Cookbook のおすすめレシピ
schroneko
7
870
Rethinking UI building strategies @ SFI 2024
letelete
0
270
Komplexe Oberflächen mit SVG und der Web Animation API
joergneumann
0
670
今、知っておきたい! 生成AIエージェントの世界
elith
3
350
効率化に挑戦してみたらモバイル開発が少し快適になった話
ryunakayama
0
130
Featured
See All Featured
"I'm Feeling Lucky" - Building Great Search Experiences for Today's Users (#IAC19)
danielanewman
221
21k
Web Components: a chance to create the future
zenorocha
305
41k
Making the Leap to Tech Lead
cromwellryan
124
8.5k
Designing with Data
zakiwarfel
96
4.8k
RailsConf & Balkan Ruby 2019: The Past, Present, and Future of Rails at GitHub
eileencodes
125
32k
KATA
mclloyd
15
12k
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
78
42k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
6
1.5k
Build your cross-platform service in a week with App Engine
jlugia
225
17k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
322
20k
How to train your dragon (web standard)
notwaldorf
73
5.2k
A Modern Web Designer's Workflow
chriscoyier
689
190k
Transcript
不要になったAWSリソースを 削除してサービス障害起こした話 Tomoki Inoue 2023/01/26
• Name: Tomoki Inoue • Join: 2019.3~ • Position: Backend
Engineer • Twitter: @_tmkinoue 自己紹介
経歴 ~2019 : PHP, JavaScript 2019 : PHP, JavaScript, AWS
2020 : Golang, AWS, Kubernetes (Amazon EKS) 2021 : Golang, AWS, Kubernetes (Amazon EKS) 2022 : React Native, AWS, Kubernetes (Amazon EKS)
不要になったと思っていたAWSリソースを 削除してサービス障害起こした話
背景
Feb. 2020 icon by Icons8
Feb. 2020 FE team BE team icon by Icons8
Feb. 2020 ~ BE team icon by Icons8
2021 Summer icon by Icons8 卒業します 異動します
2021 Summer BE team icon by Icons8
2021 Summer ~ icon by Icons8 Kubernetes Ver. Amazon EKS
end of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up
やったこと
2021 Summer icon by Icons8 Kubernetes Ver. Amazon EKS end
of support 1.17 2021.11 1.18 2022.03 1.19 2022.08 1.20 2022.11 1.21 2023.02 Ver. up New Cluster (Latest ver.)
Mar.2022 ~May.2022 icon by Icons8 Ver. up New Cluster (Latest
ver.)
AWS Cloud Amazon EKS EC2 instance contents icon by Icons8
User Internet Application Load Balancer Pod (Golang Application) Amazon CloudFront
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) $ eksctl delete cluster
結果
AWS Cloud icon by Icons8 User Internet Amazon CloudFront Amazon
EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application)
原因
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud icon by Icons8 Amazon CloudFront Amazon EKS EC2
instance contents Application Load Balancer Pod (Golang Application) Amazon EKS EC2 instance contents Application Load Balancer Pod (Golang Application) Amazon SQS Amazon DynamoDB Amazon Aurora Role
AWS Cloud $ eksctl create cluster Amazon EKS Application Load
Balancer Role Amazon EC2 AWS CloudFormation
AWS Cloud $ eksctl delete cluster Amazon EKS Application Load
Balancer Role Amazon EC2 AWS CloudFormation
新しいクラスタを作ったときは?
icon by Icons8 Ver. up New Cluster (Latest ver.)
反省と教訓
反省と教訓 • その作業、実施すると何が起こるかわかってますか? • 手順書の内容を把握しただけで理解した気になっていませんか? • レビュー、ちゃんと見て内容を理解してますか? ◦ そんな余裕はない? ◦
そこにリソースを割くことができない?
反省と教訓 • ちゃんと見てないけど担当者がいい感じにやってくれるはず • でも引き継ぎは突然やってくることもあります ◦ 退職、休職、異動... • 前任者がしっかりと仕事してくれる人だったから大丈夫? •
退職者は責任を取ってくれませんよ?
まとめ
本番環境の作業をするときはくれぐれも慎重に ☆(ゝω・)vキャピ
ご清聴ありがとうございました