Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE Lounge 20180117
Search
abnoumaru
January 17, 2018
0
6.6k
SRE Lounge 20180117
クローズな勉強会で話した資料を社外向けにしたものです
abnoumaru
January 17, 2018
Tweet
Share
More Decks by abnoumaru
See All by abnoumaru
IVRyエンジニア忘年LT大会2024 クリティカルユーザージャーニーの整理
abnoumaru
0
170
ゆるSRE勉強会 #8 組織的にSREが始まる中で意識したこと
abnoumaru
2
1.3k
3-shake SRE Tech Talk #10 LLMのO11yに触れる
abnoumaru
2
10k
マイクロサービスの現場からプラットフォームエンジニアリングの可能性を探る!
abnoumaru
2
9.4k
SLOいつ決めましょう?
abnoumaru
5
2.3k
あなたらしくSRE(公開用)
abnoumaru
4
7.6k
IDCFクラウドを使ってどこまでチューニングできるか試してみた
abnoumaru
0
210
AWS認定ソリューションアーキテクトを受けた話
abnoumaru
1
1.9k
Featured
See All Featured
Building Applications with DynamoDB
mza
91
6.1k
Visualization
eitanlees
146
15k
Java REST API Framework Comparison - PWX 2021
mraible
28
8.3k
Designing for humans not robots
tammielis
250
25k
Code Reviewing Like a Champion
maltzj
520
39k
Gamification - CAS2011
davidbonilla
80
5.1k
Docker and Python
trallard
42
3.1k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5k
The World Runs on Bad Software
bkeepers
PRO
65
11k
Automating Front-end Workflow
addyosmani
1366
200k
Visualizing Your Data: Incorporating Mongo into Loggly Infrastructure
mongodb
44
9.3k
Navigating Team Friction
lara
183
15k
Transcript
MSP ʹ͓͚ΔτΠϧݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB
Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
ࣗݾհ • ৽ଔ2ೖࣾ6 • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •
͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
ϋʔτϏʔπ • 24࣌ؒ365ͷ༗ਓࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻ΛߦʢϝσΟΞɺ EC
αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷׂ • ӡ༻άϧʔϓ 24࣌ؒ365ͷোରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲ऀʣ • ٕज़։ൃࣨ ࣾαʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾڭɺࣾશମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺͳ͍ • SRE ຊΛʢ༁ͯ͠ʣྠಡ͢Δ෦׆͋Δ • ͍ͬͯΔ͜ͱඇৗʹ͍ۙͷͰؔ৺ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
Managed Service Provider • αʔϏεʹਵͯ͠ൃੜ͢ΔۀΛ·ͱΊ͚ͯෛ͏ҰछͷΞτιʔγϯά • αʔόӡ༻ • αʔόͷख •
OSϛυϧΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷࢹ • োରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365ͷిϝʔϧʹΑΔαϙʔτ • ։ൃऀͰͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (
@abnoumaru ) 6
SRE ͷओͳৼΔ͍͔ΒݟͨϋʔτϏʔπ • ιϑτΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ୲Ͱͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͋Δ • ࣾͷγεςϜվળ • γεςϜΤϯδχΞϦϯά
• ओʹͬͯΔ • τΠϧݮ • ӡ༻ͷݱτΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
τΠϧ αʔϏεʹؔͷ͋Δ࡞ۀͷɺҎԼͷ͍ͣΕ͔ʹͯ·Δͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ
• ظతՁΛ࣋ͨͳ͍ • αʔϏεʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
τΠϧͷྫ • ଟ͋Δ • ఆྫλεΫ • ࢹઃఆ • υΩϡϝϯτ࡞ •
… • Nagios ͔Β௨͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • ΞϥʔτݮʹτΠϧݮͱݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
τΠϧͰͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫεՃ • …
• ਓؒͷஅΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭஅͷ༰ϓϩάϥϜͰॲཧͰ͖Δ߹ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏߹γεςϜΞϥʔτͷઃܭ͔Βݟ͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
τΠϧΛूΊΔ • ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
• ۀؔ࿈ͷ߹ࢥཱ͍ͬͨਓ͕ͬͯΔ • ֤࡞ۀͷ࣌ؒɾසͷચ͍ग़͠ • ӡ༻ۀͷଟ͍ӡ༻άϧʔϓ࡞ۀͷใΛચ͍ग़ͨ͠ • ༏ઌͲͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ͘͠ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
• ӡ༻ͱΤϯδχΞͰײࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ༰ bot ͔Β Redmine ʹొ͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়ࢹͰ֬ೝ • ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔͢ •
SRE ຊͰݴ͑ 12. Troubleshooting ͷτϦΞʔδʹͯ·Γͦ͏ • ఆ࣌֬ೝ • ͍߹Θͤӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19
ۀͷτΠϧݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌͳ͍ͷͰجຊతʹͯ͢ଈ࣌༰֬ೝʢࢲͷνʔϜʣ • ׂΓࠐΈ • ༰ʹΑͬͯௐ͋Δ •
ස͕ߴ͍߹ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟʢ͍·Θͤͳ͍͜ͱ͕͋ΔͷͭΒ͍ʣ • සͷߴ͍ࣾͷ࡞ۀ • ιϑτΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔɺߏͷҟͳΔαʔό 2000 ͕૬ख • Nagios ʴ
Cacti ΛखಈͰߋ৽ • ෳͰαʔό͕૿͑ΔΑΓॳΊͯ৮Δڥͷࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ѲڥͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱཧ͞Ε͍ͯΔ • ԽͷͨΊʹ༻ҙ͞Εͨෳͷ Nagios Λߋ৽ • ࣾͰࢹઃఆπʔϧΛ࡞͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷࢹ • Nagios 2 ʢʴςετ༻
1 ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯཧɺ ຊ൪αʔόίϛοτɺҰͣͭϩάΠϯͯ͠ө • ݱࡏͷࢹ • Nagios 2 ʢʴςετ༻ 1 ʣʴ Grafana ʢͷऔಘ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔੜ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪ڥө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
·ͱΊ • SRE ຊͷ༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧݮࣄྫΛհ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏ΛͬͯಓʹτΠϧΛݮ͍ͯ͠Δ
• ײ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε SRE ຊͷ༰ͱʑͷۀΛݟൺͯΈΔͱSRE Β͍͠ৼΔ͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23