Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ポストモーテム運用を支える文化と技術 / Culture and Technology Sup...
Search
Takeshi Kondo
February 09, 2023
Technology
2
2.4k
ポストモーテム運用を支える文化と技術 / Culture and Technology Supporting Postmortem Operations
https://findy.connpass.com/event/273197/
Takeshi Kondo
February 09, 2023
Tweet
Share
More Decks by Takeshi Kondo
See All by Takeshi Kondo
SRE NEXT CfP チームが語る 聞きたくなるプロポーザルとは / Proposals by the SRE NEXT CfP Team that are sure to be accepted
chaspy
1
650
Slack Platform(Deno) での RAG 実装 - LangChain(js) を使ってみた / rag-implementation-on-slack-platform-deno-experimenting-with-langchain-js
chaspy
0
180
SRE の考えをマネジメントに活かす / applying SRE ideas to management
chaspy
7
6.9k
RAGの簡易評価によるフィードバックサイクル実践 / Feedback cycle practice through simplified assessment of RAGs
chaspy
2
5.1k
定量データと定性評価を用いた技術戦略の組織的実践 / Systematic implementation of technology strategies using quantitative data and qualitative evaluation
chaspy
9
1.7k
エンジニアブランディングチームの KPI / KPI's of engineer branding team
chaspy
2
2k
「SLO Review」今やるならこうする / If I had to do the "SLO Review" again
chaspy
3
1.8k
開発者とともに作る Site Reliability Engineering / SREing with Developers
chaspy
10
8.1k
自己診断能力の獲得を目指して / Toward the acquisition of self-diagnostic skills
chaspy
1
4.9k
Other Decks in Technology
See All in Technology
大AI時代で輝くために今こそドメインにディープダイブしよう / Deep Dive into Domain in AI-Agent-Era
yuitosato
1
360
Porting PicoRuby to Another Microcontroller: ESP32
yuuu
4
410
DuckDB MCPサーバーを使ってAWSコストを分析させてみた / AWS cost analysis with DuckDB MCP server
masahirokawahara
0
1.2k
LiteXとオレオレCPUで作る自作SoC奮闘記
msyksphinz
0
600
Devinで模索する AIファースト開発〜ゼロベースから始めるDevOpsの進化〜
potix2
PRO
7
3.3k
ワールドカフェI /チューターを改良する / World Café I and Improving the Tutors
ks91
PRO
0
120
より良い開発者体験を実現するために~開発初心者が感じた生成AIの可能性~
masakiokuda
0
160
От ручной разметки к LLM: как мы создавали облако тегов в Lamoda. Анастасия Ангелова, Data Scientist, Lamoda Tech
lamodatech
0
710
ここはMCPの夜明けまえ
nwiizo
4
2.9k
Running JavaScript within Ruby
hmsk
3
320
LLM とプロンプトエンジニアリング/チューターをビルドする / LLM, Prompt Engineering and Building Tutors
ks91
PRO
1
250
【2025年度新卒技術研修】100分で学ぶ サイバーエージェントのデータベース 活用事例とMySQLパフォーマンス調査
cyberagentdevelopers
PRO
5
7.3k
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
390
[Rails World 2023 - Day 1 Closing Keynote] - The Magic of Rails
eileencodes
34
2.2k
The World Runs on Bad Software
bkeepers
PRO
67
11k
GitHub's CSS Performance
jonrohan
1030
460k
Visualization
eitanlees
146
16k
What's in a price? How to price your products and services
michaelherold
245
12k
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.2k
Mobile First: as difficult as doing things right
swwweet
223
9.6k
Docker and Python
trallard
44
3.3k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
41
2.2k
The Cult of Friendly URLs
andyhume
78
6.3k
Transcript
ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़ Takeshi Kondo / @chaspy 2023/02/07 ΠϯγσϯτʹͲ͏ରԠ͖͔ͯͨ͠ʁΈΜͳͰֶͿϙετϞʔςϜ Lunch LT
Who am I chaspy chaspy_ Engineering Manager Site Reliability and
Web Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me
લఏɿϓϩμΫτհ - ελσΟαϓϦ
ࠓ͢͜ͱ ʮϙετϞʔςϜӡ༻ʯͷલఏͱͳΔจԽͱٕज़
ࠓ͞ͳ͍͜ͱ ʮϙετϞʔςϜӡ༻ʯͦΕࣗମͷ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜӡ༻ͷݱঢ় • োൃੜޙʮϙετϞʔςϜॻ͖·͠ΐ͏ʯͷ • ؔऀͰू·ͬͯڞ༗ • ΞΫγϣϯ֤νʔϜͷΠγϡʔͱͯ͠ੵ·ΕΔ
ΧδϡΞϧʹϙετϞʔςϜ͕ߦΘΕΔ༷ࢠ ܰඍͳͷͰʮֶͼͷνϟϯεʯͱଊ͑Δ త͕ਁಁ͍ͯ͠Δ །Ұͷͱͯ͠ Slack ΧελϜ ϨεϙϯεͰ issue template ͕ग़
ͯ͘Δͷॻͨ͘ΊͷϋʔυϧΛ Լ͍͛ͯΔ…?
ੲॻ͍ͨهࣄ͕ࠓͰҾ༻͞Ε͍ͯΔ ࠓճ Findy ͞Μʹ͔͚ͯΒͬ ͨͷ͜ͷهࣄΛݟͯΒ͔ͬͨ ΒͰͨ͠🙏 2019… ʮোରԠͱϙετϞʔςϜ ελσΟαϓϦʯͰݕࡧʂ
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜӡ༻ͷྺ࢙ • Issue Template ͷ First Commit 20195݄ • ͦΕ͔ΒςϯϓϨʔτͷߋ৽΄ͱΜͲͳ͍
ϙετϞʔςϜӡ༻ͷྺ࢙ • SRE ຊ͔ΒςϯϓϨʔτྲྀ༻ • Issue Template ͷ First Commit
20195݄
ϙετϞʔςϜӡ༻ͷྺ࢙ • TTD/TTR Λه
Outline • ϙετϞʔςϜӡ༻ͷݱঢ় • ϙετϞʔςϜӡ༻ͷྺ࢙ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽ • ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ •
·ͱΊ
ϙετϞʔςϜΛࢧ͑ΔจԽ • ୭͔1ਓͷ͍ͤʹͳΒͳ͍Α͏ʹ͢Δ • Design Doc • Production Readiness Checklist
• ૉૣ͘ɺΈΜͳͰରԠ͢Δ • োରԠϑϩʔ • ো͔ΒֶͿ • ϙετϞʔςϜڞ༗ձ • ϙετϞʔςϜಡॻձ ඪ४Խ͢Δ తҙࣝͷৢ
Design Doc / Production Readiness Checklist • ʮ͏͔ͬΓʯΛඪ४Խ͢Δ • ෳਓͰϨϏϡʔ͢Δ͜ͱͰʮݸਓͷ͍ͤʯʹͮ͠Β͘͢Δ
• ϨϏϡʔͳ͠୯ಠΦϖϨʔγϣϯͰϛεΔͱͲ͏ͯ͠ݪҼ͕ݸਓʹ ͍ͯ͠·͏Ͱ͠ΐ͏ ʮProduction Readiness ελσΟαϓϦʯͰݕࡧʂ
োରԠϑϩʔ • োରԠϑϩʔɾোϨϕϧ͕ఆٛ͞Ε͍ͯΔ • Slack work fl ow ͰใࠂͰ͖Δ •
ো͔ʁͰใࠂ͢Δ͜ͱΛਪ͍ͯ͠Δ
োରԠϑϩʔ ઌͷ CircleCI ͷ݅ͷใࠂྫ ऀʹࣗಈͰϝϯγϣϯ͕ඈͿ
ϙετϞʔςϜಡॻձ • SRE νʔϜͰΦϯϘʔσΟϯάͰϙετϞʔςϜಡॻձΛ ࣮ࢪ • શ෦ಡΊͳ͍ʢ૿͑ΔʣͷͰʮ͓͢͢ΊʯϙετϞʔςϜ ΛϥϕϧͰཧ • ֶͼ͕ଟ͍ͷ
• ݱࡏͷߏཧղʹͭͳ͕Δͷ • োൃੜ࣌ͷಈ͖ͱͯ͠ࢀߟʹͳΔͷ
͓͢͢ΊϙετϞʔςϜ8બ
ϙετϞʔςϜΛࢧ͑ΔจԽ·ͱΊ • ϋʔυϧΛԼ͛Δࡉ͔ͳΈ • Issue Template, Slack custom response •
ඪ४Խ • Production Readiness Checklist, োରԠϑϩʔɺϨϕϧఆٛ • ʮֶͼͷͨΊʯͱ͍͏తҙࣝͷৢ • ࠷ॳݴ͍ଓ͚Δɾॻ͖ଓ͚Δ͔͠ͳ͍ؾ͕͠·͢ • աڈ Slack ݕࡧͯ͠ΈΔͱোʹରͯ͠ʮॻ͍ͯΒ͑·͔͢ʁʯͱΑ͓͘ئ͍͍ͯͨ͠ • ॻ͍ͨ݅ chaspy ͕Ұ൪ଟͦ͏… • ϒϩάΛॻ͘ͷޮՌ͋ͬͨͱࢥ͍·͢
ϙετϞʔςϜΛࢧ͑Δٕज़ • ॏཁͳোࣄલʹ͛ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ • దʹϦεΫΛऔΔ͜ͱ͕Ͱ͖͍ͯ·͔͢ʁ • ʮ೦ͷҝ֬ೝʯ͕؆୯ʹͰ͖ΔΑ͏ʹͳ͍ͬͯ·͔͢ʁ
ϙετϞʔςϜӡ༻Λࢧ͑Δٕज़ • ෛՙςετ • Canary Release • E2E Test Automation
• σʔλϕʔεϦετΞ
ෛՙςετ Production Readiness Checklist Ͱ Performance Risk Λಛఆͯ͠Β͍ɺ ඞཁͰ͋Ε Loadtest
ΛҊ Load Test ࣮ࢪ༰ͷ Template Requirements Λهࡌͯ͠ SRE ͱ։ൃ νʔϜͰઢΛ߹ΘͤΔ
ෛՙςετ • Gatling ͷίʔυΛॻ͍ͯςετ͕࣮ࢪͰ͖Δڥ • ςετ݁Ռ͕ PR ʹషΒΕΔ • ෛՙςετ͕ߴʹࢼߦࡨޡͰ͖Δ
Ϩϙʔτੜ
ෛՙςετ • ڥ४උ؆୯ͱݴΘͳ͍͕ɺϋʔυϧԼ͕͍ͬͯΔ • Databaseʢຊ൪͔ΒϦετΞ͢Δɻޙड़ʣ • Application (Pull Request Λ࡞ΕͰ͖Δʣ
• EKS Node Group • Test code
Canary Release • Argo Rollouts Λ׆༻ • Rails Upgrade ͳͲɺػೳมߋͳ͍͕ɺϦεΫͷߴ͍มߋʹ͏
φΠεTryͰ͢ΑͶ 1% ͔ΒϦϦʔε͠ɺΤϥʔ͕ग़ͨΒ͙͢ ͢͜ͱͰඃΛ࠷খݶʹͰ͖·ͨ͠
E2E Test Automation • ϒϩάΛݟ͍ͯͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ E2Eʯ • ݕग़͢Δෆ۩߹ͦΕͳΓʹ͋Γɺຊ൪োΛ͍Ͱ͍Δ
σʔλϕʔεϦετΞ • ͪ͜ΒৄࡉϒϩάΛ͝ཡ͍ͩ͘͞ʂ • ݕࡧʮελσΟαϓϦ σʔλϕʔεϦετΞʯ
·ͱΊ • ϙετϞʔςϜӡ༻Λࢧ͑ΔจԽͱٕज़Λհ͠·ͨ͠ • ϓϩηεɾจԽ໘ඪ४Խͱతҙࣝͷৢ͕ॏཁ • ٕज़໘ൃੜޙͷ࠶ൃࢭͷੵΈॏͶ • จԽͱٕज़ɺ྆ํ͕૬ޓʹ࿈ܞ͢Δ •
ੵΈॏͶΔ͜ͱͰʮಉ͡োʯى͖ͮΒ͘ͳΔ • ʮ৽͍͠োʯֶͼͷνϟϯεʹͳΔ
ࠓ͞ͳ͔ͬͨ͜ͱʢεϐʔΧʔτʔΫͰͤͨΒخ͍͠ʣ • োͷධՁɺϨϕϧ͚ • MTTR / MTTD ͷܭଌ • ࣄޙͷλεΫΛ͍͔ʹ։ൃΛ͠ͳ͕Β࣮ࢪ͢Δ͔
• োͱ SLI/SLO
Thank you! chaspy chaspy_ Engineering Manager Site Reliability and Web
Application Development at Recruit Co., Ltd. Takeshi Kondo https://chaspy.me