Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE Lounge 20180117
Search
abnoumaru
January 17, 2018
0
6.7k
SRE Lounge 20180117
クローズな勉強会で話した資料を社外向けにしたものです
abnoumaru
January 17, 2018
Tweet
Share
More Decks by abnoumaru
See All by abnoumaru
Road to SRE NEXT@仙台 IVRyの組織の形とSLO運用の現状
abnoumaru
1
600
IVRyエンジニア忘年LT大会2024 クリティカルユーザージャーニーの整理
abnoumaru
0
420
ゆるSRE勉強会 #8 組織的にSREが始まる中で意識したこと
abnoumaru
2
1.9k
3-shake SRE Tech Talk #10 LLMのO11yに触れる
abnoumaru
2
12k
マイクロサービスの現場からプラットフォームエンジニアリングの可能性を探る!
abnoumaru
2
11k
SLOいつ決めましょう?
abnoumaru
4
2.5k
あなたらしくSRE(公開用)
abnoumaru
5
8.2k
IDCFクラウドを使ってどこまでチューニングできるか試してみた
abnoumaru
0
250
AWS認定ソリューションアーキテクトを受けた話
abnoumaru
1
1.9k
Featured
See All Featured
Bootstrapping a Software Product
garrettdimon
PRO
307
110k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
44
2.4k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Measuring & Analyzing Core Web Vitals
bluesmoon
7
490
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
60k
Git: the NoSQL Database
bkeepers
PRO
430
65k
GraphQLとの向き合い方2022年版
quramy
46
14k
The World Runs on Bad Software
bkeepers
PRO
69
11k
Scaling GitHub
holman
459
140k
Thoughts on Productivity
jonyablonski
69
4.7k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
45
7.4k
Faster Mobile Websites
deanohume
307
31k
Transcript
MSP ʹ͓͚ΔτΠϧݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB
Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
ࣗݾհ • ৽ଔ2ೖࣾ6 • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •
͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
ϋʔτϏʔπ • 24࣌ؒ365ͷ༗ਓࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻ΛߦʢϝσΟΞɺ EC
αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷׂ • ӡ༻άϧʔϓ 24࣌ؒ365ͷোରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲ऀʣ • ٕज़։ൃࣨ ࣾαʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾڭɺࣾશମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺͳ͍ • SRE ຊΛʢ༁ͯ͠ʣྠಡ͢Δ෦׆͋Δ • ͍ͬͯΔ͜ͱඇৗʹ͍ۙͷͰؔ৺ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
Managed Service Provider • αʔϏεʹਵͯ͠ൃੜ͢ΔۀΛ·ͱΊ͚ͯෛ͏ҰछͷΞτιʔγϯά • αʔόӡ༻ • αʔόͷख •
OSϛυϧΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷࢹ • োରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365ͷిϝʔϧʹΑΔαϙʔτ • ։ൃऀͰͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (
@abnoumaru ) 6
SRE ͷओͳৼΔ͍͔ΒݟͨϋʔτϏʔπ • ιϑτΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ୲Ͱͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͋Δ • ࣾͷγεςϜվળ • γεςϜΤϯδχΞϦϯά
• ओʹͬͯΔ • τΠϧݮ • ӡ༻ͷݱτΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
τΠϧ αʔϏεʹؔͷ͋Δ࡞ۀͷɺҎԼͷ͍ͣΕ͔ʹͯ·Δͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ
• ظతՁΛ࣋ͨͳ͍ • αʔϏεʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
τΠϧͷྫ • ଟ͋Δ • ఆྫλεΫ • ࢹઃఆ • υΩϡϝϯτ࡞ •
… • Nagios ͔Β௨͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • ΞϥʔτݮʹτΠϧݮͱݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
τΠϧͰͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫεՃ • …
• ਓؒͷஅΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭஅͷ༰ϓϩάϥϜͰॲཧͰ͖Δ߹ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏߹γεςϜΞϥʔτͷઃܭ͔Βݟ͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
τΠϧΛूΊΔ • ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
• ۀؔ࿈ͷ߹ࢥཱ͍ͬͨਓ͕ͬͯΔ • ֤࡞ۀͷ࣌ؒɾසͷચ͍ग़͠ • ӡ༻ۀͷଟ͍ӡ༻άϧʔϓ࡞ۀͷใΛચ͍ग़ͨ͠ • ༏ઌͲͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ͘͠ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
• ӡ༻ͱΤϯδχΞͰײࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ༰ bot ͔Β Redmine ʹొ͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়ࢹͰ֬ೝ • ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔͢ •
SRE ຊͰݴ͑ 12. Troubleshooting ͷτϦΞʔδʹͯ·Γͦ͏ • ఆ࣌֬ೝ • ͍߹Θͤӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19
ۀͷτΠϧݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌͳ͍ͷͰجຊతʹͯ͢ଈ࣌༰֬ೝʢࢲͷνʔϜʣ • ׂΓࠐΈ • ༰ʹΑͬͯௐ͋Δ •
ස͕ߴ͍߹ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟʢ͍·Θͤͳ͍͜ͱ͕͋ΔͷͭΒ͍ʣ • සͷߴ͍ࣾͷ࡞ۀ • ιϑτΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔɺߏͷҟͳΔαʔό 2000 ͕૬ख • Nagios ʴ
Cacti ΛखಈͰߋ৽ • ෳͰαʔό͕૿͑ΔΑΓॳΊͯ৮Δڥͷࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ѲڥͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱཧ͞Ε͍ͯΔ • ԽͷͨΊʹ༻ҙ͞Εͨෳͷ Nagios Λߋ৽ • ࣾͰࢹઃఆπʔϧΛ࡞͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷࢹ • Nagios 2 ʢʴςετ༻
1 ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯཧɺ ຊ൪αʔόίϛοτɺҰͣͭϩάΠϯͯ͠ө • ݱࡏͷࢹ • Nagios 2 ʢʴςετ༻ 1 ʣʴ Grafana ʢͷऔಘ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔੜ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪ڥө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
·ͱΊ • SRE ຊͷ༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧݮࣄྫΛհ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏ΛͬͯಓʹτΠϧΛݮ͍ͯ͠Δ
• ײ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε SRE ຊͷ༰ͱʑͷۀΛݟൺͯΈΔͱSRE Β͍͠ৼΔ͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23