Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE-Lounge-8-Cookpad-Microservice-Architecture-...
Search
rrreeeyyy
March 13, 2019
Technology
5
5.5k
SRE-Lounge-8-Cookpad-Microservice-Architecture-Overview
SRE Lounge #8 でお話させていただきました
rrreeeyyy
March 13, 2019
Tweet
Share
More Decks by rrreeeyyy
See All by rrreeeyyy
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
220
An Efficient Incident Response Training with AI / SRE NEXT 2024 Sponsor Session
rrreeeyyy
1
5k
カンファレンスから見る SRE トレンド 2024 / SRE Trends from Conferences in 2024 #SRE_Findy
rrreeeyyy
4
2.4k
信頼性の育て方 / mackerel-meetup-15
rrreeeyyy
10
2.6k
SRE の歩き方・進め方 / sre-walk-through-procedure
rrreeeyyy
0
8.8k
「信頼性」を保ちつつ大規模サービスをリニューアルする / cookpad-tech-kitchen-service-embedded-sres
rrreeeyyy
11
12k
Cookpad and Prometheus
rrreeeyyy
6
21k
A survey of anomaly detection methodologies for web system
rrreeeyyy
5
1.3k
エンジニアリングをちゃんとやる あるいは 人類の平和 について / wsa02-rrreeeyyy
rrreeeyyy
14
3.1k
Other Decks in Technology
See All in Technology
Prox Industries株式会社 会社紹介資料
proxindustries
0
180
AIにどこまで任せる?実務で使える(かもしれない)AIエージェント設計の考え方
har1101
3
1.2k
AIのAIによるAIのための出力評価と改善
chocoyama
0
480
CSS、JSをHTMLテンプレートにまとめるフロントエンド戦略
d120145
0
200
CIでのgolangci-lintの実行を約90%削減した話
kazukihayase
0
340
~宇宙最速~2025年AWS Summit レポート
satodesu
1
1.1k
実践! AIエージェント導入記
1mono2prod
0
130
Amazon Bedrockで実現する 新たな学習体験
kzkmaeda
1
380
CI/CDとタスク共有で加速するVibe Coding
tnbe21
0
230
Observability infrastructure behind the trillion-messages scale Kafka platform
lycorptech_jp
PRO
0
130
ユーザーのプロフィールデータを活用した推薦精度向上の取り組み
yudai00
0
470
Oracle Cloud Infrastructure:2025年6月度サービス・アップデート
oracle4engineer
PRO
1
140
Featured
See All Featured
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Testing 201, or: Great Expectations
jmmastey
42
7.5k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Build The Right Thing And Hit Your Dates
maggiecrowley
36
2.7k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
Code Review Best Practice
trishagee
68
18k
Git: the NoSQL Database
bkeepers
PRO
430
65k
Building a Scalable Design System with Sketch
lauravandoore
462
33k
Practical Orchestrator
shlominoach
188
11k
Transcript
$PPLQBE.JDSPTFSWJDF "SDIJUFDUVSF0WFSWJFX ؙحؙػحس吳䒭⠓爡䪮遭鿇43&ؚٕ٦ف し䊛畆㣕 !SSSFFFZZZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
!SSSFFFZZZ !SSSFFFZZZ IUUQTSSSFFFZZZDPN :PTIJLBXB3ZPUB .F ˖ :PTIJLBXB3ZPUB !SSSFFFZZZ[reɪ] ˖ ؙحؙػحس吳䒭⠓爡
։ ˖ 䪮遭鿇43&ؚٕ٦ف ˖ 莆걄㚖 ˖ 0CTFSWBCJMJUZ 5JNFTFSJFTEBUBCBTF ˖ 馯 ˖ QVZP FPS[FB MFBHVFPGMFHFOET HCG ˖ TFJZV SFTUBHF XVH 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&T ˖ 秈せ +1 ˖ 㛇劤涸ח傈劤㕂ⰻד ˖ 秈せ (MPCBM ˖
؎ؘٔأ׃ֻכ傈劤㕂ⰻד 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM $PVOUSJFT-BOHVBHFT ˖
IUUQTDPPLQBEDPNVT ˖ IUUQTDPPLQBEDPNVL ˖ IUUQTDPPLQBEDPNJE ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE +1(MPCBM ˖ +BQBO ˖ IUUQTDPPLQBEDPN ˖ (MPCBM ˖ ֿך《穈כGPPUOPUFTך⚛ןח撑׃גְֻׁ
IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSFOETFBTPO IUUQTTQFBLFSEFDLDPNUBLBOBCFDIBMMFOHFTGPSHMPCBMTFSWJDFGSPNBQFSTQFDUJWFPGTSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
$PPLQBE43&Tך顑⟣眔㔲 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؎ً٦آ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
植㹋 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ 䎃׀ַو؎ؙٗ؟٦ؽأ⻉ח《穈דֹ ˖ ؝ىُص؛٦ءّٝ٥顑⟣眔㔲٥涪⸬桦זו㼎ׅ㉏겗䠐陎 ˖ 㣐ֹז⽃♧ך؝٦سك٦أ٥؟٦ؽأ㔿剣ך㉏겗ָֻׁ֮ ˖ و؎ؙٗ؟٦ؽأ⻉ָ鹌ֿהח״㣐ֹזًٔحزך❦「
˖ 涪鸞䏝ךぢ♳װ僇然ז顑⟣眔㔲٥ٔٔ٦أ鸞䏝ךぢ♳זו ˖ 㼭ְׁ؝٦سك٦أ٥؟٦ؽأזדכך兛鸐ך涪ָ遤ִ״ֲח 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.JDSPTFSWJDFTBOE$PPLQBE ˖ ♧倯דծو؎ؙٗ؟٦ؽأ⻉鹌ֿהד饯ֹ㉏겗֮ ˖ 43&ָⰋגך؟٦ؽأ鋅הأ؛٦ٕ׃זְ ˖ ؟٦ؽأ鸐⥋ך㉏겗٥鸬ꓲ涸זꥺ㹱٥〳錁庠䚍ך㉏겗 ˖ 43&ך⚺噟ך♧אכְֲֿ㉏겗ח㼎ׅ،فٗ٦ث
˖ ֿתדוך״ֲח׃ג،فٗ٦ث׃גַֹ稱➜ ˖ 4DBMBCJMJUZ "WBJMBCJMJUZך錁挿ַ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ أ؛٦ٓؽٔذ؍ծה鎉ג圫ղז⽃⡘ָ֮ ˖ ➂ ˖ ؟٦غ ˖ ؝أز 43&-PVOHF
]:PTIJLBXB3ZPUB !SSSFFFZZZ
4DBMBCJMJUZ ˖ ➂ ˖ إٕؿ؟٦ؽأ⻉鹌 ˖ ؟٦غ ˖ "VUP4DBMJOHך崞欽 ˖
؝أز ˖ Ⱏ鸐㛇湍ך4QPU⻉ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
إٕؿ؟٦ؽأ⻉ ˖ ؙحؙػحسדכ"NB[PO&$4ⵃ欽׃ג؝ٝذشךⰟ鸐㛇湍⡲גְ ˖ ،فٔ؛٦ءّٝך㹀纏װرفٗ؎חכIBLPⵃ欽׃גְ ˖ 㛇劤涸ח؟٦ؽأך涪罏ָ㹀纏剅ֹرفٗ؎דֹ״ֲחזגְ ˖ 㹀纏ؿ؋؎ٕך剅䒭חכ+TPOOFUⵃ欽׃גְ ˖
"84ך؟٦ؽأDPEFOJ[FUPPMTװ5FSSBGPSN⢪ג؝٦س⻉׃גְ ˖ 涪罏כ䗳銲ד֮ל荈歋ח13⳿ֿׅהָדֹ״ֲחזגְ IUUQTDPEFOJ[FUPPMT IUUQTHJUIVCDPNFBHMFUNUIBLP 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
؝ٝذشⰟ鸐㛇湍 ˖ &$4؎ٝأةٝأכ4QPU'MFFU"VUP4DBMJOH(SPVQד盖椚ׁגְ ˖ &$4؎ٝأةٝأךקהוכ4QPUד⹛ְגְ ˖ ؝أز٥鎘皾项彁ד㣐ֹזًٔحزָ欰תגְ ˖ ؎ٝأةٝأךٔا٦أ٥&$4؟٦ؽأך朐䡾זו然钠׃גأ؛٦ٕ،ؐز׃גְ ˖
ⰻ醡ד؟٦ؽأך朐䡾然钠דֹ8FC؝ٝا٦ٕ⡲גְ ˖ ؟٦ؽأ荈⡤ךًزؙٔأװٗ٦سغٓٝ؟ךؙٔؒأز٥ٖأهٝأךًزؙٔأזו ˖ ،فٔ؛٦ءّٝךؚٗךꠘ鋮٥嗚稊זו IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF IUUQTTQFBLFSEFDLDPNFBHMFUNUCVJMEJOHBTUFBEZFDTJOGSBTUSVDUVSF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ scheduler: { type: 'ecs', cluster: 'eagletmt', desired_count: 2, role:
'ecsServiceRole', elb_v2: { vpc_id: 'vpc-WWWWWWWW', health_check_path: '/site/sha', listeners: [ { port: 443, protocol: 'HTTPS', certificate_arn: 'arn:aws:iam::012345678901:server-certificate/hello-lb-v2.example.com', }, ], subnets: ['subnet-XXXXXXXX', 'subnet-YYYYYYYY'], security_groups: ['sg-ZZZZZZZZ'], : }, }, app: { image: 'ryotarai/hello-sinatra', : secrets: [{ name: 'MESSAGE', value_from: 'arn:aws:ssm:ap-northeast-1:012345678901:parameter/hako/hello-lb-v2/secret-message', }], }, sidecars: { front: { image_tag: 'hako-nginx', : }, }, : } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
"WBJMBCJMJUZ ˖ ؟٦ؽأך鸐⥋חֶֽ醱꧟䚍〳腉זꣲ㼰זֻׅ ˖ 4FSWJDFNFTIך㼪Ⰵ ˖ 黝ⴖזٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ ˖ ꥺ㹱ָ饯ֹהֹח.553ָ〳腉זꣲ瀉ְ״ֲחׅ ˖
.POJUPSJOH 0CTFSWBCJMJUZך然⥂זו ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ "-JGFPG$IBPT&OHJOFFSJOH4UBSUJOHXJUI3FTJMJFODF IUUQTTQFBLFSEFDLDPNJULRBMJGFPGDIBPTFOHJOFFSJOHTUBSUJOHXJUISFTJMJFODF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
4FSWJDFNFTIך《穈 ˖ ؟٦ؽأ鸐⥋ךקרⰋגד&OWPZָ欽ְגְ ˖ ،فٔ؛٦ءّٝכ➭ך؟٦ؽأח鸐⥋ׅ儗&OWPZ鸐ׅ״ֲחׅ ˖ 黝ⴖחٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ָ涪⹛ׅ✲דِ٦ؠ䕦갟ך֮♧儗涸זؒٓ٦ָ慧幾 ˖ ؟٦ؽأ鸐⥋ך➭ח鸐䌢ך5$11SPYZה׃ג⢪גְ ˖
&OWPZך$POUSPM1MBOFך؝ٝه٦طٝزה׃גDPPLQBEJUBDIP DPPLQBETET欽ְגְ ˖ ٔزٓ؎٥؟٦ؗحزـٖ٦ؕ٦ך鏣㹀זו♧⯋ד盖椚׃גְ ˖ 鏣㹀ך鎸岀חכ+TPOOFUⵃ欽׃גְ IUUQTHJUIVCDPNDPPLQBEJUBDIP IUUQTHJUIVCDPNDPPLQBETET IUUQTUFDIMJGFDPPLQBEDPNFOUSZ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
.POJUPSJOH 0CTFSWBCJMJUZפך《穈 ˖ 䖞勻ך؎ٝأةٝأך湊鋔כ׆ה֮;BCCJY⢪גְ ˖ 㢩䕎湊鋔ה׃גך4UBUVT$BLFװ鎉铂׀הך鑫稢ז"1.ה׃גך/FX3FMJDזו֮ ˖ ⟃♴ך״ֲזًزؙٔأכ1SPNFUIFVTח꧊ג鑫稢ח鋅״ֲח׃גְ ˖ ؎ٝأةٝأ٥ىسٕؐؑ،ך㛇劤涸זًزؙٔأ
node_exporter ぐىسٕؐؑ،׀הךFYQPSUFS ˖ ؝ٝذش橆㞮ך㛇劤涸זًزؙٔأ cAdvisor ٥&OWPZַ《؟٦ؽأ鸐⥋ךًزؙٔأ ˖ 1SPNFUIFVTח꧊ًزؙٔأַ،ٓ٦زׅ➬穈 ˖ +TPOOFU欽ְג3VMFך㹀纏"MFSUNBOBHFSך鏣㹀剅ֽ״ֲח׃גְ ˖ "8493BZװ7J[DFSBMזו欽ְ%JTUSJCVUFE5SBDJOH 7JTVBMJ[BUJPOך➬穈 IUUQTHJUIVCDPNSSSFFFZZZEPDLFSQSPNFUIFVTNJYJOUSFFNBTUFSFYBNQMF 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ prometheusAlerts+:: { groups+: [ { name: 'example-alerts', rules: [
{ alert: 'ExampleCriticalAlertPrometheusDownFor1min', // PromQL expr: ||| up{job="prometheus"} == 0 ||| % $._config, 'for': '1m', labels: { pager: 'sre', severity: 'critical', }, annotations: { title: '{{$labels.job}} is unreachable', description: '{{$labels.job}} at {{$labels.instance}} could not be scraped for over 1 minutes.', runbook: 'https://wiki.example.com/pages/runbook/ExampleCriticalAlertPrometheusDownFor1min' }, }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { route+: { routes+: [{ match_re: { pager:
'^(sre)$' }, receiver: 'sre-slack', routes: [{ match: { severity: 'critical' }, receiver: 'sre-pagerduty', }], }], }, }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-slack', slack_configs: [
{ send_resolved: true, channel: '#sre', }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
{ alertmanagerConfig+:: { receivers+: [ { name: 'sre-pagerduty', pagerduty_configs: [
{ service_key: '{{ template "secrets.pagerduty.sre.service_key" }}' }, ], }, ], }, } 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
תתדֹֿה٥װְֿהָ֮ ˖ 刿זإٕؿ؟٦ؽأ⻉ ˖ ぐ؟٦ؽأפךث٦ي䋆ֹ鴥4-*4-0ך㼪Ⰵ ˖ 刿ז؟٦ؽأًحءُפך《穈 ˖ 刿ז؝ٝذشⰟ鸐㛇湍ך何㊣פך《穈 ˖
H31$זוך31$'SBNFXPSLפך《穈 ˖ 0CTFSWBCJMJUZ%JTUSJCVUFE5SBDJOHפך《穈 ˖ $IBPT&OHJOFFSJOHפך《穈 ˖ 婍גְٖؖء٦橆㞮ךٌتٝ⻉ ˖ 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ
8FBSFIJSJOH ˖ IUUQTDPPLQBEKPCT ˖ IUUQTDPPLQBEXPSLBCMFDPN 43&-PVOHF ]:PTIJLBXB3ZPUB !SSSFFFZZZ