Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amaz...
Search
Shinji Fujimoto
October 08, 2016
Technology
3
2.6k
Amazon Elasticsearch Serviceを利用したAWSのログ活用 /amazones-aws-integration
Developers.IO 2016 in FUKUOKA
Shinji Fujimoto
October 08, 2016
Tweet
Share
More Decks by Shinji Fujimoto
See All by Shinji Fujimoto
AWS のサービスを活用して CI/CD #akibaaws
shinjifujimoto
0
17k
Amazon Elasticsearch Service の使いドコロ in 福岡
shinjifujimoto
1
680
Amazon Elasticsearch Service の使いドコロ
shinjifujimoto
3
52k
AWSのマネージドサービスを活用したログ可視化
shinjifujimoto
1
3k
はじめてのIoT
shinjifujimoto
0
4.8k
Beats開発の始め方 #cmdevio
shinjifujimoto
2
2.2k
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructure:2024年9月度サービス・アップデート
oracle4engineer
PRO
0
360
Oracle GoldenGate 23ai 導入Tips
oracle4engineer
PRO
1
210
小さな勉強会の始め方、広げ方、あるいは友達の作り方 / How to Start, Grow, and Build Connections with Small Study Groups
ar_tama
2
1.1k
マーケットプレイス版Oracle WebCenter Content For OCI
oracle4engineer
PRO
2
200
KDD2024参加報告
cyberagentdevelopers
PRO
1
210
HashHub会社案内「なぜ今、パブリックブロックチェーンに賭けるのか」
hashhub
3
75k
Vista FinderMx
jtes
0
160
【shownet.conf_】ローカル5Gを活用したウォーキングツアーの体感向上
shownet
PRO
0
250
[JAWS-UG GameTech] 第6回 各種事例紹介_18TRIPにおけるAWSサービスを活用した負荷テスト・障害テスト
naoto_yasuda
0
110
入門 バックアップ
ryuichi1208
15
4.7k
Hazard pointers with reference counter
ennael
PRO
0
110
CVE alive
ennael
PRO
0
370
Featured
See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
6
230
Atom: Resistance is Futile
akmur
261
25k
Debugging Ruby Performance
tmm1
73
12k
Intergalactic Javascript Robots from Outer Space
tanoku
268
27k
CoffeeScript is Beautiful & I Never Want to Write Plain JavaScript Again
sstephenson
158
15k
We Have a Design System, Now What?
morganepeng
49
7.1k
The World Runs on Bad Software
bkeepers
PRO
65
11k
The Cult of Friendly URLs
andyhume
76
6k
Teambox: Starting and Learning
jrom
131
8.7k
Thoughts on Productivity
jonyablonski
67
4.2k
Principles of Awesome APIs and How to Build Them.
keavy
126
17k
Infographics Made Easy
chrislema
239
18k
Transcript
Amazon Elasticsearch Service Λར༻ͨ͠AWSͷϩά׆༻ Developers.IO in FUKUOKA 2016/10/08 ౻ຊ ਅ࢘
Profile • Who ౻ຊ ਅ࢘ Ϋϥεϝιουגࣜձࣾ • Work AWSͷಋೖࢧԉɺϒϩά •
Interest Elastic Stack
ΞδΣϯμ • ϩάͷ׆༻ • AWSʹ͓͚Δϩά׆༻ • Amazon Elasticsearch Serviceʹ͍ͭͯ •
AWSʹ͓͚ΔϩάՄࢹԽɾੳͷϢʔεέʔε
ϩάͷ׆༻ͱʁ
͋Δͷӡ༻ݱ Α͋͘ΔWebγεςϜ ɾ ɾ ɾ Ϣʔβʔ ͍ͳ͊ɻɻ -# 8FC"1 %#
͋Δͷӡ༻ݱ Ϣʔβʔ γεςϜཧऀ %#୲ "104୲ /8୲ Կ͔શମతʹϖʔδ͕͍Έ͍ͨͳΜ͚ͩͲ ΈΜͳௐͯ
͋Δͷӡ༻ݱ γεςϜཧऀ ϩάʹΤϥʔग़ྗͳ͠ ϨεϙϯελΠϜѱ͍ 42-ͷΫΤϦͳ͍ ݁ہݪҼɾɾʁ ɾ ɾ ɾ -#
8FC"1 %#
ϩά׆༻ ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
ϩά࣮ͷ • ΞϓϦέʔγϣϯϩάͷ࣮ܗࣜΛҙ͍ࣝͯ͠·͔͢ • Plain Text • CSV/TSV • JSON
• LTSV
AWSʹ͓͚Δϩά׆༻
AWSͷϩάऩू ऩू ू ɾɾɾ ɾɾɾ ՄࢹԽ ੳ ࢹ ௨
AWSͷϩάऩू S3ͷग़ྗ S3 Bucket CloudTrail Redshift ELB CloudFront Config S3
EMR Data Pipeline Billing
AWSͷϩάऩू CloudWatch Logsͷग़ྗ EC2 (Agent) ECS API Gateway CloudTrail VPC
Flow Log Lambda CloudWatch Logs IoT
AWSͷϩάऩू CloudTrailͷग़ྗ ɹɹɹ59αʔϏεʹରԠʢ2016/10/07ݱࡏʣ http://docs.aws.amazon.com/awscloudtrail/latest/userguide/cloudtrail-supported-services.html ɾɾɾ ଞଟ CloudTrail
AWSͷϩάऩू AWSͷΠϕϯτग़ྗ EC2 RDS CodeDeploy ElastiCache Redshift CloudFormation SWF OpsWorks
Management Console AWS
AWSͷϩάऩू AWS Config ͦͷଞ
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ɾɾɾ ՄࢹԽ ੳ ଞଟ
AWSͷϩάू $MPVE8BUDI-PHT ϩάετϦʔϜͱͯ͠ू 4 ϑΝΠϧͱͯ͠ू
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάࢹɾ௨ $MPVE8BUDI-PHT Ωʔϫʔυࢹ $MPVE8BUDI ݅ʹΑΔݕ 4/4 ௨ -BNCEB ΧελϚΠζ௨
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
AWSͷϩάՄࢹԽɾੳ 2VJDL4JUF &MBTUJDTFBSDI4FSWJDF
AWSͷϩά׆༻ ऩू ू ࢹ ௨ ՄࢹԽ ੳ ଞଟ
Amazon Elasticsearch Service
Elasticsearch • LuceneϕʔεͷOSSશจݕࡧΤϯδϯ • ߴ֦ுੑɾߴՄ༻ੑ • WebAPIʹΑΔσʔλૢ࡞ʢCRUDɺSearchʣ • पลπʔϧͷॆ࣮
Kibana • ElasticsearchͷσʔλΛՄࢹԽ • ϦΞϧλΠϜੳ • ڧྗͳݕࡧʗूܭ • μογϡϘʔυԽ
Elasticsearch/Kibanaͷߏ • Elasticsearch on EC2 • Elastic Cloud • Amazon
Elasticsearch Service
Elasticsearch on EC2 ࣗલͰOS͔Βཧ͢Δ • ϝϦοτ • ࣗ༝ • ηΩϡϦςΟੑ
• σϝϦοτ • ӡ༻ɺอक
Elastic Cloud Elastic͕ࣾఏڙ͢ΔElasticsearch as a service • ϝϦοτ • ӡ༻ɺอक
• ༻ϓϥάΠϯ • σϝϦοτ • ίετ͕ଞʹൺͯߴ͍
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
• ߏஙͷ؆ུԽ • ϚϧνAZஔ • Master / Data • αΠζࢦఆ
• ӡ༻ͷ؆ུԽ • ਫฏ/ਨεέʔϧ • োͷࣗಈݕ • ࣗಈϊʔυަ • ࣍εφοϓγϣοτ • CloudWatchʹΑΔϦιʔεϞχλϦϯά AWSͷϚωʔδυαʔϏε Availability Zone 1 Availability Zone 2 Master Master Data Data Index1 Rep1 Index2 Rep2 Snapshot Monitoring
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
AWSαʔϏεͱͷ࿈ܞ AWS IoT CloudWatch Logs Kinesis Firehose AWS Lambda ηϯαʔσʔλͷ
ՄࢹԽ ϩάσʔλͷՄࢹԽ ϩάͷશจݕࡧ ετϦʔϜσʔλͷ όοϑΝϦϯά ΤϥʔϋϯυϦϯά ࣗ༝ͳར༻ αϯϓϧίʔυͷఏڙ S3 → Amazon ES DynamoDB → Amazon ES
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
IAMϙϦγʔʹΑΔΞΫηε੍ޚ IPΞυϨεʹΑΔ੍ޚʗAWSॺ໊ʹΑΔ੍ޚ
Amazon Elasticsearch Service AWS͕ఏڙ͢ΔElasticsearchͷϚωʔδυαʔϏε • AWSϚωʔδυαʔϏε • AWSαʔϏεͱͷ࿈ܞ͕Մೳ • IAMϙϦγʔʹΑΔΞΫηε੍ޚ
• ࣗ༝͕গͳ͍
Ͱ͖ͳ͍͜ͱ • ࣗ༝ͳElasticsearchͷόʔδϣϯબ • VPCରԠ • ϓϥάΠϯͷΠϯετʔϧ • Dynamic Script
• Ϣʔβʔࣙॻ • ϝϞϦεϫοϓ • Ұ෦ͷElasticsearch API • TCP Transport • Elasticࣾͷαϙʔτ • TileMap
Ϣʔεέʔε
Ϣʔεέʔε̍ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ
Կ͕Γ͔͔ͨͬͨʁ • ϘτϧωοΫՕॴͷಛఆ • LB ? • Web/AP ? •
DB ? • ϘτϧωοΫॲཧͷಛఆ • ͲͷURLύε ? • ͲͷSQLΫΤϦ ?
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 ɾ ɾ ɾ Ϣʔβʔ -#
8FC"1 %#
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ Ϣʔβʔ &-# &$ 3%4 γεςϜཧऀ &4 ूɾՄࢹԽɾੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ εέδϡʔϦϯάʁ ETLॲཧʁ ΤϥʔϋϯυϦϯάʁ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ IUUQE 3%4GPS.Z42- ग़ྗઌ 4 'JMF 5BCMFPS'JMF ΞΫηεํ๏
"84"1* ϑΝΠϧΞΫηε 42-ΫΤϦ PS "84"1* ܗࣜ εϖʔε۠Γ ΧελϚΠζՄ ϚϧνϥΠϯ λΠϛϯά ຖPSຖ ϦΞϧλΠϜ ϦΞϧλΠϜ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ EC2 Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &$ &4 $MPVE8BUDI • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ
• ΤϥʔϋϯυϦϯά • όοϑΝϦϯά • Ճίετͳ͠ -PHTUBTIɺ'MVFOUEͷར༻ $MPVE8BUDI-PHTͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ϩάࢹ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ ELB Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • EC2ͷෛՙ • ίετ &-# 4 &-# 4 -BNCEB
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ RDS for MySQL Amazon ES
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &$ &4 &4 • ϝοηʔδύʔγϯά • χΞϦΞϧλΠϜ • ΤϥʔϋϯυϦϯά
• όοϑΝϦϯά -PHTUBTIɺ'MVFOUEͷར༻ "84-BNCEBͷར༻ • ϝοηʔδύʔγϯά • EC2ͷෛՙ • ίετ 3%4 -BNCEB 3%4 $MPVE8BUDI &WFOU
Demo
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &-# &$ 3%4 &4 -BNCEB -BNCEB 4 $MPVE8BUDI &WFOU
͋Δͷӡ༻ݱΛAWSʹஔ͖͑
ύϑΥʔϚϯεղੳ • ֤Ϧιʔεʹ͓͚Δॲཧ࣌ؒΛՄࢹԽ • ॲཧ୯Ґͷॲཧ࣌ؒΛՄࢹԽ • URLύε୯ҐͷύϑΥʔϚϯεੳ • SQLΫΤϦ୯ҐͷύϑΥʔϚϯεੳ •
EC2୯ҐͷύϑΥʔϚϯεੳ
Ϣʔεέʔε̍.̑ ɹ͋Δͷӡ༻ݱͷ ɹύϑΥʔϚϯεੳ ɹʢServerless൛ʣ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ "1*(BUFXBZ -BNCEB 3%4 &4 -BNCEB -BNCEB $MPVE8BUDI
Ϣʔεέʔε̎ ɹCloudTrailՄࢹԽʹΑΔ ɹෆਖ਼ΞΫηεੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE8BUDI -PHT $MPVE5SBJM ઃఆʹΑΔ࿈ܞ $MPVE8BUDI-PHTͷઃఆʹΑΓࣗಈੜ ιʔείʔυ"84͕ఏڙ
ෆਖ਼ΞΫηεੳ • IAMϢʔβʔͷෆਖ਼ϩάΠϯ • IAMϢʔβʔɺIAMϩʔϧͷෆਖ਼ར༻ • ະ༻IAMϢʔβʔɺIAMϩʔϧͷચ͍ग़͠ • 1िؒΛ͑ͨՄࢹԽ
Ϣʔεέʔε̏ ɹCloudWatchϝτϦΫεͷ ɹظؒอଘɺ౷߹ੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 $MPVE8BUDI -PHTUBTIɺ'MVFOUEͱʹ ϓϥάΠϯ͕ఏڙ͞Ε͍ͯΔ &$
CloudWatchϝτϦΫε • 2िؒΛ͑ΔϝτϦΫεͷอ࣋ • CloudWatchҎ֎ͷϝτϦΫεͱͷൺֱ • CloudWatchʹͳ͍ԁνϟʔτɺ ΤϦΞνϟʔτͳͲͷՄࢹԽ • Ҏ֎ͷѻ͍
• ॊೈͳ݅ʹΑΔՄࢹԽ • ଟ࣍ݩͷՄࢹԽ
Ϣʔεέʔε̐ ɹCloudFrontϩάՄࢹԽʹ ΑΔΞΫηεղੳ
͋Δͷӡ༻ݱΛAWSʹஔ͖͑ &4 -BNCEB $MPVE'SPOU $MPVE'SPOUͷઃఆʹΑΓ ϩάΛ4ग़ྗ &-#ͱಉ͡Α͏ʹ࣮Մ 4 &$ -PHTUBTIɺ'MVFOUEͰ؆୯ʹ࣮Մ
CloudFrontͷϩάՄࢹԽ • ਓؾϖʔδੳ • Ωϟογϡώοτੳ • ΞΫηεҬੳ • ར༻ऀͷ/OSੳ •
ෆਖ਼ΞΫηεੳ ʢࢀߟURLʣ http://dev.classmethod.jp/server-side/elasticsearch/cloudfront-log-to-kibana/
·ͱΊ • AWSͷαʔϏεͷଟ͘ϩάΛग़ྗ͢Δ • ϩάͷతอͰͳ͘ɺ׆༻ • Elasticsearch + KibanaʹΑΓॊೈͳՄࢹԽΛ࣮ݱ •
Amazon Elasticsearch ServiceͰ͋Εɺ $20/month͔Β࢝ΊΒΕ·͢