Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ELKstackとAthenaの素敵な関係
Search
遊
February 10, 2017
Technology
0
960
ELKstackとAthenaの素敵な関係
2017/02/10 JAWS-UG#9で発表した資料になります
遊
February 10, 2017
Tweet
Share
More Decks by 遊
See All by 遊
applibotのDevOpsを支える terraform/packer
gacharu
1
3.3k
ゲームのインフラ6年やっててよく聞かれること
gacharu
1
1.2k
Other Decks in Technology
See All in Technology
Jr. Championsになって、強く連携しながらAWSをもっと使いたい!~AWSに対する期待と行動~
amixedcolor
0
190
分布で見る効果検証入門 / ai-distributional-effect
cyberagentdevelopers
PRO
4
700
独自ツール開発でスタジオ撮影をDX!「VLS(Virtual LED Studio)」 / dx-studio-vls
cyberagentdevelopers
PRO
1
180
IaC運用を楽にするためにCDK Pipelinesを導入したけど、思い通りにいかなかった話
smt7174
1
110
Oracle Base Database Service 技術詳細
oracle4engineer
PRO
5
49k
よくわからんサービスについての問い合わせが来たときの強い味方 Amazon Q について
kazzpapa3
0
220
「最高のチューニング」をしないために / hack@delta 24.10
fujiwara3
21
3.4k
AWSコンテナ本出版から3年経った今、もし改めて執筆し直すなら / If I revise our container book
iselegant
15
4k
Datachain会社紹介資料(2024年11月) / Company Deck
datachain
3
16k
来年もre:Invent2024 に行きたいあなたへ - “集中”と“つながり”で楽しむ -
ny7760
0
470
Aurora_BlueGreenDeploymentsやってみた
tsukasa_ishimaru
1
130
サイバーエージェントにおける生成AIのリスキリング施策の取り組み / cyber-ai-reskilling
cyberagentdevelopers
PRO
2
200
Featured
See All Featured
Why Our Code Smells
bkeepers
PRO
334
57k
Save Time (by Creating Custom Rails Generators)
garrettdimon
PRO
27
790
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
4
290
Scaling GitHub
holman
458
140k
How to train your dragon (web standard)
notwaldorf
88
5.7k
Fireside Chat
paigeccino
32
3k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
37
1.8k
VelocityConf: Rendering Performance Case Studies
addyosmani
325
24k
BBQ
matthewcrist
85
9.3k
Writing Fast Ruby
sferik
626
61k
GitHub's CSS Performance
jonrohan
1030
460k
Building Adaptive Systems
keathley
38
2.2k
Transcript
& - , T U B D L ͱ "
U I F O B ͷ ૉ ఢ ͳ ؔ JAWS-UGԣ #9 - Operational Excellence 2017/02/10 ଜ ༡
˞ ຊ εϥ Π υ & - ,
T U B D L ͱ " N B [ P O " U I F O B Λ ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ Ͱ ͋ Γ · ͤ Μ ͢ ͍ · ͤ Μ ʂ ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ͋ Γ · ͤ Μ
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏
͔ ͍ ͠ Ό ͍ Μ ͤ ͍
ͭ ɿ ͓ ͱ ͜ Ϩ ϕ ϧ ɿ ) 1 . 1 ࣗ ݾ հ ࣾ ձ ਓ ྺ d ɹ % $ Ͱ ࢹ Φϖ Ϩ ʔ λ ʔ d ɹ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ d / 0 8 ɹ ɹ ג ࣜ ձ ࣾ " 1 1 - * # 05 "8 4 ྺ ͳ · ͑ ɹ ɹ ɿ ଜ ɹ ༡ ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO
None
None
&-,TUBDL "NB[PO"UIFOB FMBTUJDTFBSDI LJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓͖ͯ͘͜ͱ ·ͱΊ
࣍
& - , T U B D L • elastic
search + log stash + kibana = elk stack • L ͷ෦ʢσʔλೖʣfluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ εϥΠυͰ૯শͱͯͬͯ͠·͢ɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢
FMBTUJDTFBSDI LJCBOB Ͱͷϩάࢀর
Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ͏ϩάelasticsearch • ֤αʔόͷϩάΛfluentdͰूɺೖ • ݸʑͷαʔόͷΈʹ͔͠Βͳ͍ϩάΛͳ͘͢ త͋Γ
• ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ
$ V T U P N - 0 (
& - #
3 % 4
F M B T U J D T F B
S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλϑΝΠϧͱͯ͠ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3backup • ྔͷଟ͍ϩάʢ110GBҎ্ʣஷ·Δϩά3Ͱindexຖআ • ҎલɺظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ͘ͳͬͯ͠·ͬͨ • ͦͦͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ༰ྔΛऔΒͳ͍͚ͯ͘ͳ͍ͨΊظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨͭҰिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓrundeck͞ΜͰຖࣗಈ࣮ߦཧɻS3ʹอଘɻ
• indexআޙͷௐࠪ • ݟ͍ͨϩά΄ΜͷҰ෦ • snapshotͷϦετΞͰ͖ͳ͍߹ •
ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ
• elasticsearchʹೖΕΔͷͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ)
• ٸͳґཔʹରԠ͖͠Εͳ͍
ٹ ੈ ओ
ࣄ ྫ ͍ ߹ Θ ͤ ௐ ࠪ
ͷ ҝ ɺ ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
T O B Q T I P U ͏
ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
4͔Βϩά (# Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍
ٹ ੈ ओ
\ EBUFz UISFBEll MFWFM*/'0 IBOEMFSll 64&3@*%ll
WFSTJPOll 4&44*0/@*%ll OB[P NFTTBHFl63-IUUQ 1BSBNFUFS\dུd^ IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ
$3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@ AEBUFAUJNFTUBNQ AUISFBEATUSJOH AMFWFMATUSJOH AIBOEMFSATUSJOH
A64&3@*%ATUSJOH AWFSTJPOATUSJOH A4&44*0/@*%ATUSJOH AOB[PATUSJOH ANFTTBHFATUSJOH AIPTUOBNFATUSJOH 308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞
4&-&$5 '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ
݁Ռ DTWͰͷ%-Մೳ
Β͘Β͘ڞ༗
ࣄ ྫ 4 ͷ ΞΫ η ε
ϩ ά ͕ ݟ ͨ ͍
%FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Βૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ ઃఆ͠·͢
MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW
ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC >
BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF
MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ
%&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊
ٹ ੈ ओ
$3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU #VDLFU0XOFSTUSJOH #VDLFUTUSJOH 3FRVFTU%BUF5JNFTUSJOH 3FNPUF*1TUSJOH
3FRVFTUFSTUSJOH 3FRVFTU*%TUSJOH 0QFSBUJPOTUSJOH ,FZTUSJOH 3FRVFTU63*@PQFSBUJPOTUSJOH 3FRVFTU63*@LFZTUSJOH 3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH )551TUBUVTTUSJOH &SSPS$PEFTUSJOH #ZUFT4FOUTUSJOH 0CKFDU4J[FTUSJOH 5PUBM5JNFTUSJOH 5VSO"SPVOE5JNFTUSJOH 3FGFSSFSTUSJOH 6TFS"HFOUTUSJOH 7FSTJPO*ETUSJOH 308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?> <?> aa< aa> <?> <?> <?> <?> <?> aaa <?> <?> c<?> aaa c<> <?> <?> <?> <?> <?> <?> a<?a> a <?> -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞
4&-&$5SFNPUFJQ SFRVFTUFS DPVOU DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ
SFRVFTUFS ΫΤϦ࣮ߦ
SFNPUFJQSFRVFTUFS DPVOU BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS
BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS dུd BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ݁Ռ
·ͱΊ
· ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ AthenaͰݟΔͨΊʹ~ΛS3ʹ͘Α͏ʹͯ͠…ͱ͔Ͱͳ͘ɺ ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹରͱݴ͑Δɻ AWSͷαʔϏε(S3 /
CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩάɺ ௐ͚ࠪͩͳΒ͜ΕͰेɻ • elasticsearch͔Βআͯ͠S3ʹϩάϑΝΠϧ͋ΕௐࠪOK ظؒݟΕΔ༷ʹ͢ΔͨΊʹ αʔόεϖοΫΛ্͓͛ͯ͘ඞཁແ͘ͳͬͨɻ ˠ ຊʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ ɹ→ snapshotͪΌΜͱऔΖ͏+͋Δఔظؒ࣋ͬͯͯΑ͔ͬͨ(ল)
· ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana •
S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ Ͱ ͠ ͨ
ޚ ਗ਼ ௌ ͋ Γ ͕ ͱ ͏ ͝
͟ ͍ · ͠ ͨ