Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
SRE Lounge 20180117
Search
abnoumaru
January 17, 2018
0
6.4k
SRE Lounge 20180117
クローズな勉強会で話した資料を社外向けにしたものです
abnoumaru
January 17, 2018
Tweet
Share
More Decks by abnoumaru
See All by abnoumaru
マイクロサービスの現場からプラットフォームエンジニアリングの可能性を探る!
abnoumaru
1
4.3k
SLOいつ決めましょう?
abnoumaru
4
1.1k
あなたらしくSRE(公開用)
abnoumaru
3
5.3k
IDCFクラウドを使ってどこまでチューニングできるか試してみた
abnoumaru
0
170
AWS認定ソリューションアーキテクトを受けた話
abnoumaru
1
1.8k
Featured
See All Featured
ピンチをチャンスに:未来をつくるプロダクトロードマップ #pmconf2020
aki_iinuma
88
46k
Teambox: Starting and Learning
jrom
129
8.5k
Fontdeck: Realign not Redesign
paulrobertlloyd
77
5k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
19
7.1k
Optimising Largest Contentful Paint
csswizardry
14
2.6k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
24
1.7k
How to Create Impact in a Changing Tech Landscape [PerfNow 2023]
tammyeverts
30
1.8k
Product Roadmaps are Hard
iamctodd
PRO
46
10k
The Mythical Team-Month
searls
217
42k
Happy Clients
brianwarren
93
6.5k
Fantastic passwords and where to find them - at NoRuKo
philnash
40
2.6k
Unsuck your backbone
ammeep
664
57k
Transcript
MSP ʹ͓͚ΔτΠϧݮ גࣜձࣾϋʔτϏʔπ ΤϯδχΞϦϯάάϧʔϓ ӡ༻ΤϯδχΞ Ѩ෦ وথʢ @abnoumaru ʣ HB
Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 1
ࣗݾհ • ৽ଔ2ೖࣾ6 • ΠϯϑϥΤϯδχΞ • झຯ • ͓ञɺԻָɺөըɺμʔπɺᓏࡔ46 •
͜͜࠷ۙྑ͘৮Δ • Plesk ɺKubernetes • ࣥචΛͨ͠ͷͰͦͷ͏ͪ FB ʹॻ͘ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 2
ϋʔτϏʔπ • 24࣌ؒ365ͷ༗ਓࢹͷ MSP ࣄۀ • ओʹ Web γεςϜͷӡ༻ΛߦʢϝσΟΞɺ EC
αΠτɺήʔϜɺϝʔϧɾɾɾʣ • ϋʔτϏʔπͷΤϯδχΞͷׂ • ӡ༻άϧʔϓ 24࣌ؒ365ͷোରԠɺӡ༻ɾอक࡞ۀɺݚम • ΤϯδχΞϦϯάάϧʔϓ ॳظߏஙɺҠઃɺࠜຊରԠɺӡ༻ɾอक࡞ۀʢҊ݅୲ऀʣ • ٕज़։ൃࣨ ࣾαʔϏεͷ։ൃɺ৽ٕज़ͷௐࠪɾڭɺࣾશମͷٕज़ྗఈ্͛Λܭըɾ࣮ࢪ • SRE ͱ͍͏෦ॺͳ͍ • SRE ຊΛʢ༁ͯ͠ʣྠಡ͢Δ෦׆͋Δ • ͍ͬͯΔ͜ͱඇৗʹ͍ۙͷͰؔ৺ߴ͍ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 3
Managed Service Provider • αʔϏεʹਵͯ͠ൃੜ͢ΔۀΛ·ͱΊ͚ͯෛ͏ҰछͷΞτιʔγϯά • αʔόӡ༻ • αʔόͷख •
OSϛυϧΣΞͷΠϯετʔϧɾઃఆ • ӡ༻தͷࢹ • োରԠ • ΩϟύγςΟϓϥϯχϯάɾύϑΥʔϚϯενϡʔχϯά • 24࣌ؒ365ͷిϝʔϧʹΑΔαϙʔτ • ։ൃऀͰͳ͍͕σϓϩΠ͞ΕͨΞϓϦʹ͍ͭͯݴٴ͢Δ͜ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 4
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
Φʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 5
MSP ! SRE HB Lounge 2018/01/17 - Takaaki Abe (
@abnoumaru ) 6
SRE ͷओͳৼΔ͍͔ΒݟͨϋʔτϏʔπ • ιϑτΣΞΤϯδχΞϦϯά • ΞϓϦ։ൃ୲Ͱͳ͍ͷͰগͳ͍ɺؾ͍ͮͨஈ֊Ͱݴٴ͢Δ͜ͱ͋Δ • ࣾͷγεςϜվળ • γεςϜΤϯδχΞϦϯά
• ओʹͬͯΔ • τΠϧݮ • ӡ༻ͷݱτΠϧͷࢁ • Φʔόʔϔου • ଞͷاۀಉ༷ʹ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 7
Site Reliability Engineering • ιϑτΣΞΤϯδχΞϦϯά • γεςϜΤϯδχΞϦϯά • τΠϧݮ •
ͦͷଞɺΦʔόʔϔου HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 8
τΠϧ αʔϏεʹؔͷ͋Δ࡞ۀͷɺҎԼͷ͍ͣΕ͔ʹͯ·Δͷ • ख࡞ۀͰ͋Δ • ܁Γฦ͞ΕΔ • ࣗಈԽͰ͖Δ • ઓज़తͰ͋Δ
• ظతՁΛ࣋ͨͳ͍ • αʔϏεʹରͯ͠O(n)Ͱ͋Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 9
τΠϧͷྫ • ଟ͋Δ • ఆྫλεΫ • ࢹઃఆ • υΩϡϝϯτ࡞ •
… • Nagios ͔Β௨͞ΕΔΞϥʔτϝʔϧʹରͯ͠ରԠใࠂΛߦ͍ͬͯΔͷͰ ӡ༻͔Β͢ΔͱτΠϧʁ • ΞϥʔτݮʹτΠϧݮͱݴ͑Δʁ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 10
τΠϧͰͳ͍ • ߃ٱతͳมߋ • ࠜຊղܾͷͨΊͷνϡʔχϯά • RDBMS ͷΠϯσοΫεՃ • …
• ਓؒͷஅΛඞཁͱ͢Δ࡞ۀ • ͨͩ͠࡞ۀ͕ෳࡶ͔ͭஅͷ༰ϓϩάϥϜͰॲཧͰ͖Δ߹ τΠϧͱݴͬͯྑͦ͞͏ • ʢ͜͏͍͏߹γεςϜΞϥʔτͷઃܭ͔Βݟ͢ඞཁ͕͋Δʣ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 11
τΠϧΛूΊΔ • ʑͷΞϥʔτ֬ೝ • zatsu ʹ Gitlab ͷ Issue ΛཱͯΔ
• ۀؔ࿈ͷ߹ࢥཱ͍ͬͨਓ͕ͬͯΔ • ֤࡞ۀͷ࣌ؒɾසͷચ͍ग़͠ • ӡ༻ۀͷଟ͍ӡ༻άϧʔϓ࡞ۀͷใΛચ͍ग़ͨ͠ • ༏ઌͲͷ͘Β͍վળͷݟࠐΈ͕͋Δ͔·ͱΊ͍ͯͨ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 12
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 13
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 14
Slack ͷ Thread Ͱ૬ஊ • #alert-extermination ͱ͍͏෦ʹසൃΞϥʔτΛूΊΔ • ӡ༻άϧʔϓ͘͠ΤϯδχΞ͕සൃͯ͠ΔͱࢥͬͨΒ POST
• ӡ༻ͱΤϯδχΞͰײࣝΛدͤूΊͯରԠΛܾΊΔ • Thread ͷ༰ bot ͔Β Redmine ʹొ͞ΕΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 15
ʑͷΞϥʔτ֬ೝ • োൃੜ • ӡ༻άϧʔϓ͕ঢ়گௐࠪɾҰ࣍ରԠ • ϝʔϧͰରԠใࠂૹ৴ • සൃΞϥʔτ Slack
ͷ Thread Ͱ૬ஊ • ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 16
ରԠใࠂͱఆ࣌֬ೝΛຖே֬ೝ • ରԠใࠂ • ݱঢ়ࢹͰ֬ೝ • ʹΑΔ • ରԠ͕ඞཁ͔൱͔ɺඞཁͳΒͲ͏͢Δ͔͢ •
SRE ຊͰݴ͑ 12. Troubleshooting ͷτϦΞʔδʹͯ·Γͦ͏ • ఆ࣌֬ೝ • ͍߹Θͤӡ༻άϧʔϓ͔Βͷ૬ஊͷ͋Δ Redmine νέοτͷҰཡ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 17
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 18
HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 19
ۀͷτΠϧݮࣄྫ • ઌํґཔ • ઌํʹΑΔ༏ઌͳ͍ͷͰجຊతʹͯ͢ଈ࣌༰֬ೝʢࢲͷνʔϜʣ • ׂΓࠐΈ • ༰ʹΑͬͯௐ͋Δ •
ස͕ߴ͍߹ࣗಈԽ͢Δ • طଘͷαʔό্ͰॲཧΛ͢Δ͜ͱ͕ଟ͍ͷͰ bash Python Ͱ εΫϦϓτΛ༻ҙ͢Δ͜ͱ͕ଟʢ͍·Θͤͳ͍͜ͱ͕͋ΔͷͭΒ͍ʣ • සͷߴ͍ࣾͷ࡞ۀ • ιϑτΣΞΤϯδχΞϦϯάͰղܾ͍ͯ͠Δ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 20
ࢹઃఆͷࣗಈԽ • ݱࡏ 150 Ҋ݅ఔɺߏͷҟͳΔαʔό 2000 ͕૬ख • Nagios ʴ
Cacti ΛखಈͰߋ৽ • ෳͰαʔό͕૿͑ΔΑΓॳΊͯ৮Δڥͷࢹઃఆ͕ଟ͍ • ಈ࡞͍ͯ͠ΔαʔϏεΛ֬ೝ͠ͳ͕Β NRPE ɺ Nagios Λઃఆ • ѲڥͷରԠʹ͕͔͔࣌ؒΔ • Subversion Ͱཧ͞Ε͍ͯΔ • ԽͷͨΊʹ༻ҙ͞Εͨෳͷ Nagios Λߋ৽ • ࣾͰࢹઃఆπʔϧΛ࡞͢Δ͜ͱͰղܾ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 21
happo1 • Golang Ͱॻ͔Ε͍ͯΔ • ैདྷͷࢹ • Nagios 2 ʢʴςετ༻
1 ʣʴ Cacti • खಈͰ NRPE ͷߋ৽ɺख࡞ۀͰςεταʔόͷߋ৽ɺ Subversion Ͱόʔδϣϯཧɺ ຊ൪αʔόίϛοτɺҰͣͭϩάΠϯͯ͠ө • ݱࡏͷࢹ • Nagios 2 ʢʴςετ༻ 1 ʣʴ Grafana ʢͷऔಘ mackerel-agent ʣ • ॳظઃఆ࣌ʹΫϩʔϦϯάͯ͠ඞཁͳ yaml Λ͋Δఔੜ • εΫϦϓτΛ 3 ճ࣮ߦ͢Δͱຊ൪ڥө 1 https://github.com/heartbeatsjp/happo-agent HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 22
·ͱΊ • SRE ຊͷ༰Λަ͑ͯ MSP ʹ͓͚ΔτΠϧݮࣄྫΛհ • τΠϧΛूΊΔ͍͔ͭ͘ͷϧʔνϯ͕͋Δ • దࡐదॴʹ͍Ζ͍Ζͳํ๏ΛͬͯಓʹτΠϧΛݮ͍ͯ͠Δ
• ײ • Կ͔͠ΒͷαʔϏεΛఏڙ͍ͯ͠Δ IT اۀͰ͋Ε SRE ຊͷ༰ͱʑͷۀΛݟൺͯΈΔͱSRE Β͍͠ৼΔ͍͕݁ߏ ݟ͔ͭΔ HB Lounge 2018/01/17 - Takaaki Abe ( @abnoumaru ) 23