Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ポストモーテム運用を支える文化と技術 / Culture and Technology Sup...
Search
Takeshi Kondo
February 09, 2023
Technology
2
2.7k
ポストモーテム運用を支える文化と技術 / Culture and Technology Supporting Postmortem Operations
https://findy.connpass.com/event/273197/
Takeshi Kondo
February 09, 2023
Tweet
Share
More Decks by Takeshi Kondo
See All by Takeshi Kondo
SRE NEXT CfP チームが語る 聞きたくなるプロポーザルとは / Proposals by the SRE NEXT CfP Team that are sure to be accepted
chaspy
2
1.4k
Slack Platform(Deno) での RAG 実装 - LangChain(js) を使ってみた / rag-implementation-on-slack-platform-deno-experimenting-with-langchain-js
chaspy
0
230
SRE の考えをマネジメントに活かす / applying SRE ideas to management
chaspy
7
7.7k
RAGの簡易評価によるフィードバックサイクル実践 / Feedback cycle practice through simplified assessment of RAGs
chaspy
2
5.5k
定量データと定性評価を用いた技術戦略の組織的実践 / Systematic implementation of technology strategies using quantitative data and qualitative evaluation
chaspy
9
1.9k
エンジニアブランディングチームの KPI / KPI's of engineer branding team
chaspy
2
2.2k
「SLO Review」今やるならこうする / If I had to do the "SLO Review" again
chaspy
3
2k
開発者とともに作る Site Reliability Engineering / SREing with Developers
chaspy
10
8.4k
自己診断能力の獲得を目指して / Toward the acquisition of self-diagnostic skills
chaspy
1
5.2k
Other Decks in Technology
See All in Technology
Django's GeneratedField by example - DjangoCon US 2025
pauloxnet
0
160
なぜテストマネージャの視点が 必要なのか? 〜 一歩先へ進むために 〜
moritamasami
0
240
バイブスに「型」を!Kent Beckに学ぶ、AI時代のテスト駆動開発
amixedcolor
2
590
Create Ruby native extension gem with Go
sue445
0
130
Snowflake Intelligenceにはこうやって立ち向かう!クラシルが考えるAI Readyなデータ基盤と活用のためのDataOps
gappy50
0
280
普通のチームがスクラムを会得するたった一つの冴えたやり方 / the best way to scrum
okamototakuyasr2
0
110
現場で効くClaude Code ─ 最新動向と企業導入
takaakikakei
1
260
Rustから学ぶ 非同期処理の仕組み
skanehira
1
150
Unlocking the Power of AI Agents with LINE Bot MCP Server
linedevth
0
120
AWSを利用する上で知っておきたい名前解決のはなし(10分版)
nagisa53
10
3.2k
Generative AI Japan 第一回生成AI実践研究会「AI駆動開発の現在地──ブレイクスルーの鍵を握るのはデータ領域」
shisyu_gaku
0
330
開発者を支える Internal Developer Portal のイマとコレカラ / To-day and To-morrow of Internal Developer Portals: Supporting Developers
aoto
PRO
1
480
Featured
See All Featured
Producing Creativity
orderedlist
PRO
347
40k
Embracing the Ebb and Flow
colly
87
4.8k
Product Roadmaps are Hard
iamctodd
PRO
54
11k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
52
5.6k
GraphQLの誤解/rethinking-graphql
sonatard
72
11k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Into the Great Unknown - MozCon
thekraken
40
2k
KATA
mclloyd
32
14k
Fight the Zombie Pattern Library - RWD Summit 2016
marcelosomers
234
17k
Build your cross-platform service in a week with App Engine
jlugia
231
18k
Facilitating Awesome Meetings
lara
55
6.5k
Documentation Writing (for coders)
carmenintech
74
5k
Transcript
ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़ Takeshi Kondo / @chaspy 2023/02/07 ΠϯγσϯτʹͲ͏ରԠ͖͔ͯͨ͠ʁΈΜͳͰֶͿϙετϞʔςϜ Lunch LT
Who am I chaspy chaspy_ Engineering Manager Site Reliability and
Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me
લఏɿϓϩμΫτհ - ελσΟαϓϦ
ࠓ͢͜ͱ ʮϙετϞʔςϜӡ༻ʯͷલఏͱͳΔจԽͱٕज़
ࠓ͞ͳ͍͜ͱ ʮϙετϞʔςϜӡ༻ʯͦΕࣗମͷ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜӡ༻ͷݱঢ় • োൃੜޙʮϙετϞʔςϜॻ͖·͠ΐ͏ʯͷ • ؔऀͰू·ͬͯڞ༗ • ΞΫγϣϯ֤νʔϜͷΠγϡʔͱͯ͠ੵ·ΕΔ
ΧδϡΞϧʹϙετϞʔςϜ͕ߦΘΕΔ༷ࢠ ܰඍͳͷͰʮֶͼͷνϟϯεʯͱଊ͑Δ త͕ਁಁ͍ͯ͠Δ །Ұͷͱͯ͠ Slack ΧελϜ ϨεϙϯεͰ issue template ͕ग़
ͯ͘Δͷॻͨ͘ΊͷϋʔυϧΛ Լ͍͛ͯΔ…?
ੲॻ͍ͨهࣄ͕ࠓͰҾ༻͞Ε͍ͯΔ ࠓճ Findy ͞Μʹ͔͚ͯΒͬ ͨͷ͜ͷهࣄΛݟͯΒ͔ͬͨ ΒͰͨ͠🙏 2019… ʮোରԠͱϙετϞʔςϜ ελσΟαϓϦʯͰݕࡧʂ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜӡ༻ͷྺ࢙ • Issue Template ͷ First Commit 20195݄ • ͦΕ͔ΒςϯϓϨʔτͷߋ৽΄ͱΜͲͳ͍
ϙετϞʔςϜӡ༻ͷྺ࢙ • SRE ຊ͔ΒςϯϓϨʔτྲྀ༻ • Issue Template ͷ First Commit
20195݄
ϙετϞʔςϜӡ༻ͷྺ࢙ • TTD/TTR Λه
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜΛࢧ͑ΔจԽ • ୭͔1ਓͷ͍ͤʹͳΒͳ͍Α͏ʹ͢Δ • Design Doc • Production Readiness Checklist
• ૉૣ͘ɺΈΜͳͰରԠ͢Δ • োରԠϑϩʔ • ো͔ΒֶͿ • ϙετϞʔςϜڞ༗ձ • ϙετϞʔςϜಡॻձ ඪ४Խ͢Δ తҙࣝͷৢ
Design Doc / Production Readiness Checklist • ʮ͏͔ͬΓʯΛඪ४Խ͢Δ • ෳਓͰϨϏϡʔ͢Δ͜ͱͰʮݸਓͷ͍ͤʯʹͮ͠Β͘͢Δ
• ϨϏϡʔͳ͠୯ಠΦϖϨʔγϣϯͰϛεΔͱͲ͏ͯ͠ݪҼ͕ݸਓʹ ͍ͯ͠·͏Ͱ͠ΐ͏ ʮProduction Readiness ελσΟαϓϦʯͰݕࡧʂ
োରԠϑϩʔ • োରԠϑϩʔɾোϨϕϧ͕ఆٛ͞Ε͍ͯΔ • Slack work fl ow ͰใࠂͰ͖Δ •
ো͔ʁͰใࠂ͢Δ͜ͱΛਪ͍ͯ͠Δ
োରԠϑϩʔ ઌͷ CircleCI ͷ݅ͷใࠂྫ ऀʹࣗಈͰϝϯγϣϯ͕ඈͿ
ϙετϞʔςϜಡॻձ • SRE νʔϜͰΦϯϘʔσΟϯάͰϙετϞʔςϜಡॻձΛ ࣮ࢪ • શ෦ಡΊͳ͍ʢ૿͑ΔʣͷͰʮ͓͢͢ΊʯϙετϞʔςϜ ΛϥϕϧͰཧ • ֶͼ͕ଟ͍ͷ
• ݱࡏͷߏཧղʹͭͳ͕Δͷ • োൃੜ࣌ͷಈ͖ͱͯ͠ࢀߟʹͳΔͷ
͓͢͢ΊϙετϞʔςϜ8બ
ϙετϞʔςϜΛࢧ͑ΔจԽ·ͱΊ • ϋʔυϧΛԼ͛Δࡉ͔ͳΈ • Issue Template, Slack custom response •
ඪ४Խ • Production Readiness Checklist, োରԠϑϩʔɺϨϕϧఆٛ • ʮֶͼͷͨΊʯͱ͍͏తҙࣝͷৢ • ࠷ॳݴ͍ଓ͚Δɾॻ͖ଓ͚Δ͔͠ͳ͍ؾ͕͠·͢ • աڈ Slack ݕࡧͯ͠ΈΔͱোʹରͯ͠ʮॻ͍ͯΒ͑·͔͢ʁʯͱΑ͓͘ئ͍͍ͯͨ͠ • ॻ͍ͨ݅ chaspy ͕Ұ൪ଟͦ͏… • ϒϩάΛॻ͘ͷޮՌ͋ͬͨͱࢥ͍·͢
ϙετϞʔςϜΛࢧ͑Δٕज़ • ॏཁͳোࣄલʹ͛ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ • దʹϦεΫΛऔΔ͜ͱ͕Ͱ͖͍ͯ·͔͢ʁ • ʮ೦ͷҝ֬ೝʯ͕؆୯ʹͰ͖ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ
ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ෛՙςετ • Canary Release • E2E Test Automation
• σʔλϕʔεϦετΞ
ෛՙςετ Production Readiness Checklist Ͱ Performance Risk Λಛఆͯ͠Β͍ɺ ඞཁͰ͋Ε Loadtest
ΛҊ Load Test ࣮ࢪ༰ͷ Template Requirements Λهࡌͯ͠ SRE ͱ։ൃ νʔϜͰઢΛ߹ΘͤΔ
ෛՙςετ • Gatling ͷίʔυΛॻ͍ͯςετ͕࣮ࢪͰ͖Δڥ • ςετ݁Ռ͕ PR ʹషΒΕΔ • ෛՙςετ͕ߴʹࢼߦࡨޡͰ͖Δ
Ϩϙʔτੜ
ෛՙςετ • ڥ४උ؆୯ͱݴΘͳ͍͕ɺϋʔυϧԼ͕͍ͬͯΔ • Databaseʢຊ൪͔ΒϦετΞ͢Δɻޙड़ʣ • Application (Pull Request Λ࡞ΕͰ͖Δʣ
• EKS Node Group • Test code
Canary Release • Argo Rollouts Λ׆༻ • Rails Upgrade ͳͲɺػೳมߋͳ͍͕ɺϦεΫͷߴ͍มߋʹ͏
φΠεTryͰ͢ΑͶ 1% ͔ΒϦϦʔε͠ɺΤϥʔ͕ग़ͨΒ͙͢ ͢͜ͱͰඃΛ࠷খݶʹͰ͖·ͨ͠
E2E Test Automation • ϒϩάΛݟ͍ͯͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ E2Eʯ • ݕग़͢Δෆ۩߹ͦΕͳΓʹ͋Γɺຊ൪োΛ͍Ͱ͍Δ
σʔλϕʔεϦετΞ • ͪ͜ΒৄࡉϒϩάΛ͝ཡ͍ͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ σʔλϕʔεϦετΞʯ
·ͱΊ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़Λհ͠·ͨ͠ • ϓϩηεɾจԽ໘ඪ४Խͱతҙࣝͷৢ͕ॏཁ • ٕज़໘ൃੜޙͷ࠶ൃࢭͷੵΈॏͶ • จԽͱٕज़ɺ྆ํ͕૬ޓʹ࿈ܞ͢Δ •
ੵΈॏͶΔ͜ͱͰʮಉ͡োʯى͖ͮΒ͘ͳΔ • ʮ৽͍͠োʯֶͼͷνϟϯεʹͳΔ
ࠓ͞ͳ͔ͬͨ͜ͱʢεϐʔΧʔτʔΫͰͤͨΒخ͍͠ʣ • োͷධՁɺϨϕϧ͚ • MTTR / MTTD ͷܭଌ • ࣄޙͷλεΫΛ͍͔ʹ։ൃΛ͠ͳ͕Β࣮ࢪ͢Δ͔
• োͱ SLI/SLO
Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web
Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me