Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE Lounge 20180117
Search
abnoumaru
January 17, 2018
0
6.7k
SRE Lounge 20180117
クローズな勉強会で話した資料を社外向けにしたものです
abnoumaru
January 17, 2018
Tweet
Share
More Decks by abnoumaru
See All by abnoumaru
Road to SRE NEXT@仙台 IVRyの組織の形とSLO運用の現状
abnoumaru
0
380
IVRyエンジニア忘年LT大会2024 クリティカルユーザージャーニーの整理
abnoumaru
0
330
ゆるSRE勉強会 #8 組織的にSREが始まる中で意識したこと
abnoumaru
2
1.7k
3-shake SRE Tech Talk #10 LLMのO11yに触れる
abnoumaru
2
11k
マイクロサービスの現場からプラットフォームエンジニアリングの可能性を探る!
abnoumaru
2
10k
SLOいつ決めましょう?
abnoumaru
4
2.4k
あなたらしくSRE(公開用)
abnoumaru
5
7.9k
IDCFクラウドを使ってどこまでチューニングできるか試してみた
abnoumaru
0
240
AWS認定ソリューションアーキテクトを受けた話
abnoumaru
1
1.9k
Featured
See All Featured
Art, The Web, and Tiny UX
lynnandtonic
298
20k
Fireside Chat
paigeccino
37
3.3k
Java REST API Framework Comparison - PWX 2021
mraible
29
8.5k
Exploring the Power of Turbo Streams & Action Cable | RailsConf2023
kevinliebholz
31
4.7k
Statistics for Hackers
jakevdp
798
220k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Testing 201, or: Great Expectations
jmmastey
42
7.4k
The Straight Up "How To Draw Better" Workshop
denniskardys
232
140k
Refactoring Trust on Your Teams (GOTO; Chicago 2020)
rmw
34
2.9k
Raft: Consensus for Rubyists
vanstee
137
6.8k
実際に使うSQLの書き方 徹底解説 / pgcon21j-tutorial
soudai
177
52k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
Transcript
MSP ʹ͓͚ΔτΠϧݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB
Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
ࣗݾհ • ৽ଔ2ೖࣾ6 • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •
͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
ϋʔτϏʔπ • 24࣌ؒ365ͷ༗ਓࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻ΛߦʢϝσΟΞɺ EC
αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷׂ • ӡ༻άϧʔϓ 24࣌ؒ365ͷোରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲ऀʣ • ٕज़։ൃࣨ ࣾαʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾڭɺࣾશମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺͳ͍ • SRE ຊΛʢ༁ͯ͠ʣྠಡ͢Δ෦׆͋Δ • ͍ͬͯΔ͜ͱඇৗʹ͍ۙͷͰؔ৺ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
Managed Service Provider • αʔϏεʹਵͯ͠ൃੜ͢ΔۀΛ·ͱΊ͚ͯෛ͏ҰछͷΞτιʔγϯά • αʔόӡ༻ • αʔόͷख •
OSϛυϧΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷࢹ • োରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365ͷిϝʔϧʹΑΔαϙʔτ • ։ൃऀͰͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (
@abnoumaru ) 6
SRE ͷओͳৼΔ͍͔ΒݟͨϋʔτϏʔπ • ιϑτΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ୲Ͱͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͋Δ • ࣾͷγεςϜվળ • γεςϜΤϯδχΞϦϯά
• ओʹͬͯΔ • τΠϧݮ • ӡ༻ͷݱτΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
τΠϧ αʔϏεʹؔͷ͋Δ࡞ۀͷɺҎԼͷ͍ͣΕ͔ʹͯ·Δͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ
• ظతՁΛ࣋ͨͳ͍ • αʔϏεʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
τΠϧͷྫ • ଟ͋Δ • ఆྫλεΫ • ࢹઃఆ • υΩϡϝϯτ࡞ •
… • Nagios ͔Β௨͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • ΞϥʔτݮʹτΠϧݮͱݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
τΠϧͰͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫεՃ • …
• ਓؒͷஅΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭஅͷ༰ϓϩάϥϜͰॲཧͰ͖Δ߹ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏߹γεςϜΞϥʔτͷઃܭ͔Βݟ͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
τΠϧΛूΊΔ • ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
• ۀؔ࿈ͷ߹ࢥཱ͍ͬͨਓ͕ͬͯΔ • ֤࡞ۀͷ࣌ؒɾසͷચ͍ग़͠ • ӡ༻ۀͷଟ͍ӡ༻άϧʔϓ࡞ۀͷใΛચ͍ग़ͨ͠ • ༏ઌͲͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ͘͠ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
• ӡ༻ͱΤϯδχΞͰײࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ༰ bot ͔Β Redmine ʹొ͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়ࢹͰ֬ೝ • ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔͢ •
SRE ຊͰݴ͑ 12. Troubleshooting ͷτϦΞʔδʹͯ·Γͦ͏ • ఆ࣌֬ೝ • ͍߹Θͤӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19
ۀͷτΠϧݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌͳ͍ͷͰجຊతʹͯ͢ଈ࣌༰֬ೝʢࢲͷνʔϜʣ • ׂΓࠐΈ • ༰ʹΑͬͯௐ͋Δ •
ස͕ߴ͍߹ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟʢ͍·Θͤͳ͍͜ͱ͕͋ΔͷͭΒ͍ʣ • සͷߴ͍ࣾͷ࡞ۀ • ιϑτΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔɺߏͷҟͳΔαʔό 2000 ͕૬ख • Nagios ʴ
Cacti ΛखಈͰߋ৽ • ෳͰαʔό͕૿͑ΔΑΓॳΊͯ৮Δڥͷࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ѲڥͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱཧ͞Ε͍ͯΔ • ԽͷͨΊʹ༻ҙ͞Εͨෳͷ Nagios Λߋ৽ • ࣾͰࢹઃఆπʔϧΛ࡞͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷࢹ • Nagios 2 ʢʴςετ༻
1 ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯཧɺ ຊ൪αʔόίϛοτɺҰͣͭϩάΠϯͯ͠ө • ݱࡏͷࢹ • Nagios 2 ʢʴςετ༻ 1 ʣʴ Grafana ʢͷऔಘ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔੜ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪ڥө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
·ͱΊ • SRE ຊͷ༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧݮࣄྫΛհ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏ΛͬͯಓʹτΠϧΛݮ͍ͯ͠Δ
• ײ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε SRE ຊͷ༰ͱʑͷۀΛݟൺͯΈΔͱSRE Β͍͠ৼΔ͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23