Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ELKstackとAthenaの素敵な関係
Search
遊
February 10, 2017
Technology
0
950
ELKstackとAthenaの素敵な関係
2017/02/10 JAWS-UG#9で発表した資料になります
遊
February 10, 2017
Tweet
Share
More Decks by 遊
See All by 遊
applibotのDevOpsを支える terraform/packer
gacharu
1
3.2k
ゲームのインフラ6年やっててよく聞かれること
gacharu
1
1.2k
Other Decks in Technology
See All in Technology
AIを活用した柔軟かつ効率的な社内リソース検索への取り組み
cygames
0
190
サーバー管理しないサーバーサービスManaged DevOps Pool
kkamegawa
0
130
持続可能なソフトウェア開発を支える『GitHub CI/CD実践ガイド』
tmknom
8
1.4k
不動産tech Product Night#2_AIことはじめ_GA橋本
takehikohashimoto
0
190
可視化により内部品質をあげるAIドキュメントリバース/20240910 Hiromitsu Akiba
shift_evolve
0
220
LLVM/ASMを使った有限体の高速実装
herumi
0
120
Oracle Autonomous Database:サービス概要のご紹介
oracle4engineer
PRO
1
7.1k
忙しい人のためのLangGraph概要まとめ
__ymgc__
1
190
Road to Single Activity
yurihondo
2
240
言葉は感情の近似値である。その感情と言葉の誤差を最小化しよう ~コミュニケーションにおけるアナログ/デジタル変換の課題に立ち向かう~
nktamago
0
230
Analytics-Backed App Widget Development - Served with Jetpack Glance
miyabigouji
0
620
事前準備が肝!AI活用のための業務改革
layerx
PRO
1
390
Featured
See All Featured
Building Applications with DynamoDB
mza
90
6k
個人開発の失敗を避けるイケてる考え方 / tips for indie hackers
panda_program
89
16k
Fantastic passwords and where to find them - at NoRuKo
philnash
48
2.8k
Building Your Own Lightsaber
phodgson
101
6k
Building Better People: How to give real-time feedback that sticks.
wjessup
359
19k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
36
1.7k
Navigating Team Friction
lara
183
13k
From Idea to $5000 a Month in 5 Months
shpigford
379
46k
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
23
1.7k
Code Reviewing Like a Champion
maltzj
517
39k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
0
120
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
502
140k
Transcript
& - , T U B D L ͱ "
U I F O B ͷ ૉ ఢ ͳ ؔ JAWS-UGԣ #9 - Operational Excellence 2017/02/10 ଜ ༡
˞ ຊ εϥ Π υ & - ,
T U B D L ͱ " N B [ P O " U I F O B Λ ٕ ज़ త ʹ ࿈ ܞ ͞ ͤ ͨ Ͱ ͋ Γ · ͤ Μ ͢ ͍ · ͤ Μ ʂ ͱ ͍ ͏ ͔ ٕ ज़ త ͳ ͋ Γ · ͤ Μ
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏
͔ ͍ ͠ Ό ͍ Μ ͤ ͍
ͭ ɿ ͓ ͱ ͜ Ϩ ϕ ϧ ɿ ) 1 . 1 ࣗ ݾ հ ࣾ ձ ਓ ྺ d ɹ % $ Ͱ ࢹ Φϖ Ϩ ʔ λ ʔ d ɹ ι ʔ γ ϟϧ ήʔϜ ܥ ձ ࣾ d / 0 8 ɹ ɹ ג ࣜ ձ ࣾ " 1 1 - * # 05 "8 4 ྺ ͳ · ͑ ɹ ɹ ɿ ଜ ɹ ༡ ͭ ͍ ͬ ͨ ʔ ɿ !HBDIBSJPO
None
None
&-,TUBDL "NB[PO"UIFOB FMBTUJDTFBSDI LJCBOBͰͷϩάࢀর ٹੈओ"UIFOB ӡ༻5JQTతͳ ҰԠ৮Ε͓͖ͯ͘͜ͱ ·ͱΊ
࣍
& - , T U B D L • elastic
search + log stash + kibana = elk stack • L ͷ෦ʢσʔλೖʣfluentdͩͬͨΓ͢Δͱ EFKͱͳΔΒ͍͠ɻ • elasticsearchʹೖΕͯkibanaͰݟͯΔͷͰ εϥΠυͰ૯শͱͯͬͯ͠·͢ɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ
" N B [ P O " U I
F O B • 2016/12/01 ެ։ • US East (Northern Virginia)ͱUS West (Oregon) Ͱར༻Մೳ(2017/02/09 ݱࡏ) • S3্ͷσʔλʹϑΟʔϧυΛఆٛͯ͠ ΫΤϦΛ͛ΔͱɺඵͰ݁Ռ͕ฦͬͯ͘Δɻ ˢϚδͰ ਆ͡Όͳ͍͔͢
FMBTUJDTFBSDI LJCBOB Ͱͷϩάࢀর
Ͷ Β ͍ • ͱΓ͋͑ͣௐࠪʹ͏ϩάelasticsearch • ֤αʔόͷϩάΛfluentdͰूɺೖ • ݸʑͷαʔόͷΈʹ͔͠Βͳ͍ϩάΛͳ͘͢ త͋Γ
• ͨ·ʹlogstash • ΫΤϦʹΑΔΞϥʔτઃఆʢࣗલεΫϦϓτʣ
$ V T U P N - 0 (
& - #
3 % 4
F M B T U J D T F B
S D I ӡ ༻ ϧ ʔ ϧ త ͳ • ೖΕΔσʔλϑΝΠϧͱͯ͠ॻ͖ग़͓ͯ͘͠ʢJSONʣ • dailyͰgzipͯ͠S3backup • ྔͷଟ͍ϩάʢ110GBҎ্ʣஷ·Δϩά3Ͱindexຖআ • ҎલɺظͰݟΕΔΑ͏ʹ͍͕ͯͨ͠ɺkibanaද͕͔ࣔͳΓ͘ͳͬͯ͠·ͬͨ • ͦͦͦΜͳʹաڈͷΛݟͳ͔ͬͨ • ετϨʔδ༰ྔΛऔΒͳ͍͚ͯ͘ͳ͍ͨΊظؒอ࣋ͨ͘͠ͳ͔ͬͨ • εφοϓγϣοτͱͬͨΓͱͬͯͳ͔ͬͨΓ(ũųųƅƁ • ͱ͍ͬͯͨͭҰिؒ΄ͲͰফ͍ͯͨ͠ɻ • ࠓrundeck͞ΜͰຖࣗಈ࣮ߦཧɻS3ʹอଘɻ
• indexআޙͷௐࠪ • ݟ͍ͨϩά΄ΜͷҰ෦ • snapshotͷϦετΞͰ͖ͳ͍߹ •
ेGBͷϩάΛDLͯ͠ղౚͯ͠grepͯ͠ɾɾɾ
• elasticsearchʹೖΕΔͷͦΕͳΓʹ༻ҙ͕ඞཁ • ύʔαʔ୳ͨ͠Γɺॻ͍ͨΓ • template࡞ͬͨΓ(dynamic templateͰେମରԠ)
• ٸͳґཔʹରԠ͖͠Εͳ͍
ٹ ੈ ओ
ࣄ ྫ ͍ ߹ Θ ͤ ௐ ࠪ
ͷ ҝ ɺ ա ڈ ͷ " 1 * ϩ ά ͕ Έ ͨ ͍
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ
ͦ Ε ͳ Β L J C B O B
Ͱ ݟ Ε · ͬ ͤ ʂ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
T O B Q T I P U ͏
ͳ ͍ Θ ʜ Ұिؒ΄ͲલͳΜͰ͕͢ʜ
4͔Βϩά (# Λ%-ͯ͠ղౚͯ͠HSFQ ͭΒ͍
ٹ ੈ ओ
\ EBUFz UISFBEll MFWFM*/'0 IBOEMFSll 64&3@*%ll
WFSTJPOll 4&44*0/@*%ll OB[P NFTTBHFl63-IUUQ 1BSBNFUFS\dུd^ IPTUOBNFlBQJTEGTGPN ^ ݩϩά +40/ܗࣜ
$3&"5&&95&3/"-5"#-&*'/05&9*454BQJTFSWFSBQJMPH@@ AEBUFAUJNFTUBNQ AUISFBEATUSJOH AMFWFMATUSJOH AIBOEMFSATUSJOH
A64&3@*%ATUSJOH AWFSTJPOATUSJOH A4&44*0/@*%ATUSJOH AOB[PATUSJOH ANFTTBHFATUSJOH AIPTUOBNFATUSJOH 308'03."54&3%&PSHPQFOYEBUBKTPOTFSEF+TPO4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU -0$"5*0/TMPHCVDLFUBQJTFSWFS %#5BCMF࡞
4&-&$5 '30.BQJMPH@@ 8)&3&VTFS@JE ΫΤϦ࣮ߦ
݁Ռ DTWͰͷ%-Մೳ
Β͘Β͘ڞ༗
ࣄ ྫ 4 ͷ ΞΫ η ε
ϩ ά ͕ ݟ ͨ ͍
%FW 0QT -BNCEBͰը૾ॲཧͯ͠Δ͠ "1*͔Βૢ࡞͍ͯ͠ΔͷͰɺ 4ΞΫηεϩάΈ͍ͨͰ͢ʂ XFCϖʔδೖͬͯΔΘ͚Ͱͳ͍͔ Β0/ʹͯ͠ͳ͔ͬͨΘʜ ઃఆ͠·͢
MPHHJOH0/ 0QT ϩάग़Δ·Ͱʹ͔͔࣌ؒΔͱॻ͍ͯ͋Δ ˞IUUQEPDTBXTBNB[PODPN ϩάLJCBOBͰݟΕ·͔͢Ͷʁ %&-&5&Λߦͳ͍ͬͯΔϦΫΤετ͕ݟ͍ͨ ɹ;Ή ɹFMBTUJDTFBSDIʹೖΕͯ ɹݕࡧͰ͖ΔΑ͏ʹ͠Α͏ %FW
ͯ͞FMBTUJDTFBSDIʹೖΕΔ४උ͠ͱ͔͘ɻ ͲΜͳײ͡ͷ͚ͭͩͬʁ 0QT φχίϨ IUUQEPDTBXTBNB[PODPN KB@KQ"NB[PO4MBUFTUEFW 4ΞΫηεϩάϩάܗࣜαϯϓϧ BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCFNZCVDLFU<'FC >
BEGCFEBFGCBDFEGEFEFBDGGEFDEFGCF&'&9".1-& 3&45(&57&34*0/*/((&5NZCVDLFU WFSTJPOJOH)5514$POTPMF
MPHTUBTIͰύʔεͰ͖Δ༷ʹͯ͠ʜ FMBTUJDTFBSDIʹೖΕͯʜ LJCBOBͰEBTICPBSE࡞ͬͯʜ
%&-&5&ͷϩάΛݟ͍͚ͨͩͳͷʹʜ ࣌ؒʜ͔͔Δʜͳ͊
ٹ ੈ ओ
$3&"5&&95&3/"-5"#-&*'/05&9*454T@BDDFTTMPHSFTPVSTF@CVDLFU #VDLFU0XOFSTUSJOH #VDLFUTUSJOH 3FRVFTU%BUF5JNFTUSJOH 3FNPUF*1TUSJOH
3FRVFTUFSTUSJOH 3FRVFTU*%TUSJOH 0QFSBUJPOTUSJOH ,FZTUSJOH 3FRVFTU63*@PQFSBUJPOTUSJOH 3FRVFTU63*@LFZTUSJOH 3FRVFTU63*@IUUQ1SPUPWFSTJPOTUSJOH )551TUBUVTTUSJOH &SSPS$PEFTUSJOH #ZUFT4FOUTUSJOH 0CKFDU4J[FTUSJOH 5PUBM5JNFTUSJOH 5VSO"SPVOE5JNFTUSJOH 3FGFSSFSTUSJOH 6TFS"HFOUTUSJOH 7FSTJPO*ETUSJOH 308'03."54&3%&PSHBQBDIFIBEPPQIJWFTFSEF3FHFY4FS%F 8*5)4&3%&1301&35*&4 TFSJBMJ[BUJPOGPSNBU JOQVUSFHFY <?> <?> aa< aa> <?> <?> <?> <?> <?> aaa <?> <?> c<?> aaa c<> <?> <?> <?> <?> <?> <?> a<?a> a <?> -0$"5*0/bTBXTMPH4SFTPVSTF@CVDLFU %#5BCMF ࡞
4&-&$5SFNPUFJQ SFRVFTUFS DPVOU DPVOU '30.SFTPVSTF@CVDLFU 8)&3&SFRVFTUVSJ@PQFSBUJPO%&-&5& HSPVQCZSFNPUFJQ
SFRVFTUFS ΫΤϦ࣮ߦ
SFNPUFJQSFRVFTUFS DPVOU BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS
BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS dུd BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ˎˎˎˎˎˎˎˎ BSOBXTJBNˎˎˎˎˎˎˎˎVTFSEFWFMPQFS ݁Ռ
·ͱΊ
· ͱ Ί • Amazon AthenaͰS3্ͷϩά͕ଈղੳՄೳঢ়ଶʹ AthenaͰݟΔͨΊʹ~ΛS3ʹ͘Α͏ʹͯ͠…ͱ͔Ͱͳ͘ɺ ͢ͰʹόοΫΞοϓͱ্͍ͯ͛ͯͨ͠ϩάશ͕ͯࣗಈతʹରͱݴ͑Δɻ AWSͷαʔϏε(S3 /
CloudFront / ELB / CloudTrail)͔Βॻ͖ग़͢ϩάɺ ௐ͚ࠪͩͳΒ͜ΕͰेɻ • elasticsearch͔Βআͯ͠S3ʹϩάϑΝΠϧ͋ΕௐࠪOK ظؒݟΕΔ༷ʹ͢ΔͨΊʹ αʔόεϖοΫΛ্͓͛ͯ͘ඞཁແ͘ͳͬͨɻ ˠ ຊʹඞཁͳظؒͷΈͰΑ͘ͳͬͨɻ ɹ→ snapshotͪΌΜͱऔΖ͏+͋Δఔظؒ࣋ͬͯͯΑ͔ͬͨ(ল)
· ͱ Ί • ϦΞϧλΠϜͷϩάղੳ → elasticsearch + kibana •
S3ʹ͓͍ͨϩάௐࠪ → Amazon Athena
& - , T U B D L Ͱ ͷ
ӡ ༻ Ͱ ײ ͡ ͨ ෆ ศ Λ " N B [ P O " U I F O B ͕ औ Γ আ ͍ͯ ͘ Ε ͨ ͷ Ͱ ײ ಈ Λ ڞ ༗ ͠ ͨ ͍ ͱ ͍ ͏ Ͱ ͠ ͨ
ޚ ਗ਼ ௌ ͋ Γ ͕ ͱ ͏ ͝
͟ ͍ · ͠ ͨ