Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Search
Yutaka Yawata
September 28, 2017
Technology
1
150
AWSで始めるビッグデータ分析 Developers.IO 2017 WORLD in 名古屋
Developers.IO 2017 WORLD in 名古屋 登壇資料
Yutaka Yawata
September 28, 2017
Tweet
Share
More Decks by Yutaka Yawata
See All by Yutaka Yawata
AWSで始めるビッグデータ分析 Developers.IO 2017
yuyawata
1
2.9k
AWSで作るオムニチャネル戦略実現のためのマーケティング・プラットフォーム
yuyawata
1
1.7k
Other Decks in Technology
See All in Technology
【株式会社ELYZA】|GENIAC成果報告会 自社開発モデルプレゼンテーション
elyza
1
250
開発者の定量・定性データを組み合わせて開発者体験を把握するための取り組み
ham0215
1
130
フルカイテン株式会社 採用資料
fullkaiten
0
32k
Segment Anything Model 2
tenten0727
3
700
忙しい人のためのLangGraph概要まとめ
__ymgc__
1
190
React Aria で実現する次世代のアクセシビリティ
ryo_manba
4
1.2k
プロダクトエンジニアを支えるための開発生産性向上施策
tsukakei
0
140
Developer Experienceを向上させる基盤づくりの取り組み事例集
coconala_engineer
0
150
Next.js のページ遷移を全力で止める
ypresto
6
3.2k
The XZ Backdoor Story
fr0gger
0
3.6k
サプライチェーン攻撃に備える
ryunen344
0
280
タイミーのレコメンドにおける ABテストの運用
ozeshun
1
120
Featured
See All Featured
ReactJS: Keep Simple. Everything can be a component!
pedronauck
663
120k
Web Components: a chance to create the future
zenorocha
308
42k
How To Stay Up To Date on Web Technology
chriscoyier
786
250k
The Brand Is Dead. Long Live the Brand.
mthomps
53
38k
Responsive Adventures: Dirty Tricks From The Dark Corners of Front-End
smashingmag
248
20k
Facilitating Awesome Meetings
lara
49
5.9k
Java REST API Framework Comparison - PWX 2021
mraible
PRO
27
7.4k
Build The Right Thing And Hit Your Dates
maggiecrowley
30
2.3k
Learning to Love Humans: Emotional Interface Design
aarron
270
40k
Visualization
eitanlees
142
15k
Side Projects
sachag
451
42k
[RailsConf 2023] Rails as a piece of cake
palkan
48
4.6k
Transcript
"84Ͱ࢝ΊΔϏοάσʔλੳ %FWFMPQFST*0803-%JO໊ݹ ݄ Ϋϥεϝιουגࣜձࣾീ഼๛ DNEFWJP
ࣗݾհ wീ഼๛ʢ:VUBLB:BXBUBʣ wϓϦηʔϧε୲ wݩ&31ύοέʔδͷج൫ίϯαϧ ࣗݾհ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ηογϣϯͷରऀ w"84ͰσʔλੳΛ࢝Ί͍ͨ w"84Ͱσʔλੳج൫Λߏங͢ΔͨΊͷ ϕετϓϥΫςΟεΛΓ͍ͨ w"84ͷར༻ܦݧ͕͋Δ 71$ &$ 3%4 4 FUD
͜ͷηογϣϯͰ͓͠͠ͳ͍͜ͱ wͳͥ"84͔ʁ w֤"84αʔϏεͷৄࡉͳػೳεϖοΫ wػցֶशɺ"*
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
༷ʑͳϏοάσʔλ ɨ 104σʔλ ϞόΠϧΞϓϦ ߦಈϩά ΫϦοΫ ετϦʔϜ 4/4
ηϯλʔσʔλ ɓ
σʔλੳͰར༻Ͱ͖Δ"84αʔϏε Amazon DynamoDB Amazon RDS Amazon Redshift Amazon S3
AWS Snowball AWS Lambda AWS Batch Amazon EC2 AWS DMS Amazon ES Amazon EMR Amazon Kinesis Amazon Athena Amazon CloudSearch Amazon QuickSight AWS Data Pipeline Amazon Machine Learning AWS IoT ଟछଟ༷ͳαʔϏε܈ σʔλҠߦ σʔλϕʔε ετϨʔδ ੳ ݕࡧ *P5 ίϯϐϡʔςΟϯά ػցֶश AWS Glue
ͲͷαʔϏεΛ͏͖͔ʁ Ͳ͏ͬͯ"84ʹσʔλΛΞοϓϩʔυ͢Δʁ σʔλͷՃͲ͜ͰΔʁ&$ʁ 3FETIJGUͬͨํ͕Α͍ʁ3%4Ͱେৎʁ
σʔλੳΞʔΩςΫνϟͷݪଇ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w ऩूɺલॲཧɺอଘɺੳɺՄࢹԽ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w εϧʔϓοτɺϨΠςϯγʔɺεέʔϥϏϦςΟɺ ߹ੑɺѻ͑ΔσʔλλΠϓʗσʔλߏɺ
ΧελϚΠζੑɺίετɺ্ݶ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
ϏοάσʔλύΠϓϥΠϯ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ϏοάσʔλύΠϓϥΠϯอଘ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
σʔλ4ʹू Amazon S3 Amazon EC2 AWS Aurora Amazon Redshift AWS
Lambda Amazon ES Amazon EMR Amazon CloudSearch Amazon QuickSight Amazon Athena AWS IoT Amazon Kinesis Streams Amazon Kinesis Firehose AWS DMS AWS Batch AWS DynamoDB
ϏοάσʔλύΠϓϥΠϯऩू લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ऩू ϑΝΠϧ ετϦʔϜ w ച্σʔλ w ސ٬Ϛελ w Ϛελ w
ߦಈϩά w ΫϦοΫετϦʔϜ w *P5ηϯαʔσʔλ ɏ σʔλλΠϓʹண AWS IoT Amazon S3 Amazon Kinesis Streams Amazon Kinesis Firehose … … …
ϏοάσʔλύΠϓϥΠϯલॲཧ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
લॲཧ ϦΞϧλΠϜ όον w ࣌ؒຖɺ࣍ɺि࣍ɺ w εϧʔϓοτॏࢹ w ஞ࣍ॲཧ w
ϨΠςϯγʔॏࢹ Amazon EMR Amazon EC2 AWS Lambda Amazon Kinesis Analytics ɏ ϫʔΫϩʔυʹண ƽ
ϏοάσʔλύΠϓϥΠϯੳ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ੳ σʔλߏ ΞΫηεύλʔϯ σʔλετΞ ੳ ݻఆ͞ΕͨεΩʔϚ ෳࡶͳϦϨʔγϣφϧΫΤϦɺ ଟͷϢʔβʔ͔Βͷಉ࣌ΞΫηε 3%4 େنσʔλͷूܭɾநग़
3FETIJGU 4 3FETIJGU4QFDUSVN εΩʔϚϨε γϯϓϧͳΫΤϦɺ͍ϨΠςϯγʔ %ZOBNP%# ੜσʔλʹର͢ΔΞυϗοΫੳ 4 &.3 1SFTUP *NQBMB 4QBSL "UIFOB "UIFOB શจݕࡧ &MBTUJD4FBSDI4FSWJDF $MPVE4FBSDI ɏ σʔλߏɺΞΫηεύλʔϯʹண
ϏοάσʔλύΠϓϥΠϯՄࢹԽ લॲཧ ऩू อଘ ੳ ՄࢹԽ |
ՄࢹԽ ఏڙํ๏ ՄࢹԽπʔϧ ηϧϑ#* "NB[PO2VJDL4JHIU ఆܕϨϙʔτ ΧελϜ8FCΞϓϦέʔγϣϯ w %ZOBNP%# -BNCEB
"1*(BUFXBZ 4 w 3%4 &$ ϩάੳͷμογϡϘʔυ "NB[PO&MBTUJD4FBSDI4FSWJDF ,JCBOB ɏ σʔλͷఏڙํ๏ʹண͢Δ w "84αʔϏεͷબࢶগͳ͍αʔυύʔςΟͷ πʔϧݕ౼ ǟ ǟ
͜ͷηογϣϯʹ͍ͭͯ ύΠϓϥΠϯͰߟ͑Δσʔλੳج൫ɹ σʔλੳϦϑΝϨϯεΞʔΩςΫνϟ
ϦϑΝϨϯεΞʔΩςΫνϟ wǡ ച্ੳ wΦϯϓϨϛεͷ%#ʹ͋Δച্σʔλ wҎԼͷੳڥΛఏڙ wੜσʔλʹର͢ΔΞυϗοΫੳ wηϧϑ#* wఆܕϨϙʔτ
ച্ੳσʔλऩू ੜσʔλ σʔλ4ʹू σʔλΞοϓϩʔυ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳੜσʔλͷΞυϗοΫੳ Amazon Athena ੜσʔλ σʔλΞοϓϩʔυ ΞυϗοΫੳ "UIFOBͰ ΞυϗοΫੳ ǟ
Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳσʔλมɺܗ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon EMR σʔλΞοϓϩʔυ ΞυϗοΫੳ
σʔλมɺܗ &.3Ͱσʔλมɺܗ ؆қͳॲཧͰ͋Ε&$Ͱࡁ·ͤΔ͜ͱ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ όοναʔόʔͰ $01:ίϚϯυΛ࣮ߦ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ3FETIJGUऔࠐΈ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ σΟϝϯγϣϯ ςʔϒϧ 3FETIJGU 4QFDUSVN ϑΝΫτςʔϒϧ4ʹɻ 4QFDUSVNͰΞΫηε ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳ#*πʔϧͰੳɺՄࢹԽ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon QuickSight Amazon Redshift
όοναʔόʔ Amazon EMR σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ 4QFDUSVN 2VJDL4JHIUͰՄࢹԽɻ ͦͷଞ#*πʔϧ༻Մೳ ՄࢹԽ ǟ Internet ੳ୲ऀ (ਓʙेਓ)
ച্ੳϨϙʔτఏڙ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet σʔλϚʔτΛ3%4ʹอଘ͠ɺ ϨϙʔτӾཡऀʹ։์
ച্ੳશମ૾ Amazon Athena ੜσʔλ ՃࡁΈσʔλ Amazon Redshift όοναʔόʔ Amazon
EMR 4QFDUSVN Amazon RDS for PostgreSQL dblink NBUFSJBMJ[FEWJFX Ϩϙʔτදࣔ αʔόʔ ੳ୲ऀ (ਓʙेਓ) σʔλϩʔυ ίϚϯυ࣮ߦ σʔλΞοϓϩʔυ ΞυϗοΫੳ σʔλมɺܗ ϨϙʔτӾཡऀ (100ਓʙ) Amazon QuickSight ǟ ՄࢹԽ Internet Internet
ϦϑΝϨϯεΞʔΩςΫνϟ w | 8FCථγεςϜ w8FC͔ΒථσʔλΛड͚͚ wථσʔλΛɺ wϦΞϧλΠϜʹूܭ wޙ#*πʔϧͰੳ
8FCථγεςϜථσʔλड ୲ऀ Amazon Kinesis Streams Internet ථσʔλड Internet "1*(BUFXBZΛϓϩΩγʹͯ͠
,JOFTJT4USFBNʹථσʔλΛొ Amazon API Gateway Amazon CloudFront ථϖʔδ4 $MPVE'SPOUͰ৴ Amazon S3 ථը໘
8FCථγεςϜ%#ͷొ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Internet ථσʔλ -BNCEBͰ%ZOBNP%#ʹ ථσʔλΛొ Internet Amazon CloudFront ථσʔλड Amazon S3 Amazon API Gateway ථը໘
8FCථγεςϜϦΞϧλΠϜूܭ ୲ऀ Amazon Kinesis Streams Amazon Kinesis Analytics Amazon
Kinesis Streams AWS Lambda Amazon DynamoDB AWS Lambda ථσʔλड ථσʔλूܭ αϚϦσʔλొ ථσʔλొ Internet αϚϦσʔλ ථσʔλ ूܭͨ͠αϚϦσʔλΛ %ZOBNP%#ʹొ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ ,JOFTJT"OBMZUJDTͰ ථσʔλΛϦΞϧλΠϜूܭ
8FCථγεςϜΞʔΧΠϒ ୲ऀ Amazon Kinesis Streams Amazon DynamoDB AWS Lambda
ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ Internet Amazon CloudFront Amazon API Gateway Amazon S3 ථը໘ શථσʔλ,JOFTJT'JSFIPTFܦ༝Ͱ 4ʹΞʔΧΠϒ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
8FCථγεςϜσʔλࢀর ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN ථσʔλΛϦΞϧλΠϜʹࢀর Internet Amazon API Gateway Amazon CloudFront ूܭ݁Ռ4ʹΞοϓϩʔυ͠ $MPVE'SPOU͔Β৴ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ Amazon S3 ථཧը໘
8FCථγεςϜશମ૾ ୲ऀ ౦ژϦʔδϣϯ Amazon Kinesis Streams Amazon DynamoDB AWS
Lambda ථσʔλड ථσʔλొ Amazon Kinesis Firehose S3 Internet αϚϦσʔλ ථσʔλ AWS Lambda %ZOBNP%# 4USFBN Internet Amazon API Gateway Amazon CloudFront Amazon S3 ථཧը໘ Amazon S3 ථը໘ Amazon Kinesis Analytics Amazon Kinesis Streams ථσʔλूܭ AWS Lambda αϚϦσʔλొ
·ͱΊ wσʔλੳΛύΠϓϥΠϯͰߟ͑Δ w"84ͷ֤αʔϏε͕Կʹ࠷దԽ͞Ε͍ͯΔ͔ʁ·ͨ ࠷దԽ͞Εͨ݁ՌͷτϨʔυΦϑԿ͔ʁΛཧղ͢Δ w֤αʔϏεͷಛΛཧղ্ͨ͠ͰɺύΠϓϥΠϯ্ͷ ͦΕͧΕͷδϣϒʹ࠷దͳαʔϏεΛબ͢Δ
׀幠耮ָ֮הֲ׀ְׂת׃կ
None