Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
データサイエンスのためのAWSの使い方入門
Search
Show Murai
July 21, 2017
Technology
1
930
データサイエンスのためのAWSの使い方入門
データサイエンスのためのAWSの使い方入門
サポーターズCoLab勉強会 Jul 20 2017
村井 翔太朗
Show Murai
July 21, 2017
Tweet
Share
More Decks by Show Murai
See All by Show Murai
システムを構築するときに 気をつける10のこと
showmurai
4
140
監視入門!監視で困ってませんか?
showmurai
0
300
AWS vs GCP 今から作るならどちらがいいの 20180330
showmurai
1
1.3k
システムを構築するときに 気をつける10のこと
showmurai
0
1.5k
KubernetesでCLIを快適を使いたい
showmurai
0
2k
AWS vs GCP 今から作る ならどっちがいいの!?
showmurai
52
83k
Other Decks in Technology
See All in Technology
BLADE: An Attempt to Automate Penetration Testing Using Autonomous AI Agents
bbrbbq
0
320
Why App Signing Matters for Your Android Apps - Android Bangkok Conference 2024
akexorcist
0
130
Adopting Jetpack Compose in Your Existing Project - GDG DevFest Bangkok 2024
akexorcist
0
110
Terraform Stacks入門 #HashiTalks
msato
0
360
Taming you application's environments
salaboy
0
190
SSMRunbook作成の勘所_20241120
koichiotomo
3
160
インフラとバックエンドとフロントエンドをくまなく調べて遅いアプリを早くした件
tubone24
1
430
Evangelismo técnico: ¿qué, cómo y por qué?
trishagee
0
360
DynamoDB でスロットリングが発生したとき/when_throttling_occurs_in_dynamodb_short
emiki
0
250
Lambdaと地方とコミュニティ
miu_crescent
2
370
Application Development WG Intro at AppDeveloperCon
salaboy
0
190
マルチモーダル / AI Agent / LLMOps 3つの技術トレンドで理解するLLMの今後の展望
hirosatogamo
37
12k
Featured
See All Featured
Six Lessons from altMBA
skipperchong
27
3.5k
RailsConf 2023
tenderlove
29
900
Bash Introduction
62gerente
608
210k
Sharpening the Axe: The Primacy of Toolmaking
bcantrill
38
1.8k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
The Art of Delivering Value - GDevCon NA Keynote
reverentgeek
8
890
KATA
mclloyd
29
14k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
How GitHub (no longer) Works
holman
310
140k
Code Review Best Practice
trishagee
64
17k
Keith and Marios Guide to Fast Websites
keithpitt
409
22k
Into the Great Unknown - MozCon
thekraken
32
1.5k
Transcript
σʔλαΠΤϯεͷͨΊͷ AWSͷ͍ํೖ αϙʔλʔζCoLabษڧձ Jul 20 2017 ଜҪ ᠳଠ࿕
ࣗݾհ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ
※ຊͷൃදݸਓͷݟղͰ͋Γɺॴଐ͢Δ৫ͷެࣜݟղͰ͋Γ·ͤΜ
͓·͑AWSৄ͍͠ͷʁ
ਓฒΈʹͬͯΔ(ͣ) • AWSྺɿ3͘Β͍ • AWSͰ͖ͳαʔϏεɿRedshiftɺLambda • AWSೝఆɿAWS SAA • Re:Invent
2015 ࢀՃ͠·ͨ͠
ΊͪΌͪ͘Όৄ͋͘͠Γ·ͤΜ ઌAWS SAPམͪ·ͨ͠…
օ༷ʹ࣭͍ͤͯͩ͘͞͞
σʔλαΠΤϯςΟετͷํ
σʔλੳ͠ͳ͍͚Ͳ σʔλੳج൫ͷߏஙΛ͢Δํ
ͳΜͰͦͦࠓͷ͠ σʔλαΠΤϯεͱAWSͳͷʁ
༑ਓ ࠷ۙɺσʔλαΠΤϯεςΟετʹͳͬͨΘ ͦΕͦΕɺ͓څྉ͗ΐ͏͞Μ͏ͯΔΜΖ ଜҪ ϕϯνϟʔ͔Β·ͩ·ͩ͜Ε͔Βɻ ·ͩGPUͱ͔αʔόʔΛങ͏͓ۚͳ͍͠ɻ Ͱֶशͤ͞Δͷʹ͔͔ΔͷΑͶʔ ͋ΕɺΫϥυͬͯͳ͍ͷʁ Ϋϥυͬͯͬͨ͜ͱͳ͍͚Ͳߴ͍Ͱ͠ΐ ͋ͱ͏ͷͦ͠͏͠
ωοτϫʔΫͱ͔ߏஙͱ͔Ͱ͖ͳ͍͠ ͦΜͳ͜ͱ͍͜͠ͱ͋ΒΜͰ ͦ͏ͳͷʁ ΄ͳࠓίʔώʔ͓͝Δ͔Β͓͑ͯ͠ (130ԁͰͪΐͬͱ…)
ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹੳ༻ͷΫΤϦΛ͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗͍ͨ͠ ͬͯݴΘΕͨͷ͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜ΕͨͿΜΫϥυҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥυͳͷʁ نʹΑΓ·͕͢Ϋϥυͷ΄͏͕
։ൃεϐʔυ͕͘ɺίετ҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜͬͨΒ ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…ԶʹλεΫ͕) ͜Ε͚ͩͱΕ·Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞Β͍·͢Θɻ
ͦΜͳਓੈͷதʹ ͪΐͬͱ͚͍ͩΔͷͰͳ͍͔ͱɻ
ຊͷରऀ • Ϋϥυ͍͍ͨσʔλαΠΤϯςΟετͷํ • σʔλαΠΤϯςΟετ(Ά͍ਓ)ʹ ج൫ͷߏஙΛ͓ئ͍͞Εͨํ • AWSͬͨ͜ͱͳ͍ํ • ࣗࣾ/ֶߍͰσʔλੳج൫Λ࣋ͬͯͳ͍ํ
ࠓ͓͢Δ͜ͱ • AWSͱ • σʔλαΠΤϯεͱ • AWSαʔϏεհ • Amazon EC2
• Amazon EMR • Amazon S3 • Amazon Redshift • ΫϥυͰࣄނΛ͙ͨΊʹ
AWSͱʁ
AWS(Amazon Web Service) ͱ ΫϥυίϯϐϡʔςΟϯάαʔϏε →Ϩϯλϧαʔόʔͷ͍ͭ͢͝ ʮΠϯϑϥ͕WebαʔϏεʹͳͬͨʯ AWS ʹ͍ͭͯ https://aws.amazon.com/jp/about-aws/
AWSੈքதʹ͋Δ • Ϧʔδϣϯ • େͳͲཧతʹΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ •
ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿɺཱ ※࣮ࡍʹެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
AWSͰग़དྷΔ͜ͱ(΄ΜͷҰ෦) • ؆୯ʹߏஙͰ͖Δίϯϐϡʔτ(αʔόʔ) • EC2 • ΫϦοΫ͚ͩͰग़དྷΔσʔλϕʔεͷߏங • RDS •
੍ݶແ͠ͷσʔλετϨʔδ • S3 • ετϦʔϜॲཧ • Kinesis (Stream)
σʔλαΠΤϯεͱʁ σʔλΛऩूɺੳɺར༻ͯ͠༗ޮ׆༻͢Δٕज़·ֶͨ
σʔλαΠΤϯςΟετͱ σʔλαΠΤϯεྗɺ σʔλΤϯδχΞϦϯάྗΛϕʔεʹ σʔλ͔ΒՁΛग़͠ɺ Ϗδωε՝ʹ͑Λग़͢ ϓϩϑΣογϣφϧ ※͜͜ͰʮϏδωεʯͱࣾձʹʹཱͭҙຯͷ͋Δ׆ಈશൠΛࢦ͢ Ұൠࣾஂ๏ਓσʔλαΠΤϯςΟετڠձ σʔλαΠΤϯςΟετͷϛογϣϯɺεΩϧηοτɺఆٛɺεΩϧϨϕϧΛൃද http://www.datascientist.or.jp/news/2014/pdf/1210.pdf
ͳͥσʔλαΠΤϯεͰ AWS(Ϋϥυ)Λ͏ͷ͔
σʔλੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλΣΞϋεʹੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ •
ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔͨ͘͘͠ͳ͍ • ߏͳͲࣗ༝͕ߴ͍
ඞཁͳ࣌ʹ ඞཁͳϦιʔε͚ͩΛ֬อ͠ ඞཁͳ͚ͩੳ͍ͨ͠ Ϋϥυ͕࠷దͰ͢Αʂ
σʔλੳ͢Δࡍʹ AWSͷͲͷαʔϏε͕͑Δ͔
հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift ԾαʔόߏஙαʔϏε
εέʔϥϒϧͰ ͚ͬͨͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อͰ͖Δ ੩తίϯςϯπͷWebϗεςΟϯάՄೳ AWS͕ఏڙ͢Δ શϚωʔδυͷσʔλΣΞϋε SQL͕࣮ߦͰ͖ϖλόΠτنͷ σʔλΛѻ͑Δ
Amazon EC2
Amazon EC2ͱ • ΫϥυίϯϐϡʔςΟϯά • ͍ΘΏΔԾαʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ
• ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛ͳΠϯελϯελΠϓ Amazon EC2 ͱ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html
Amazon EC2ͷҙ • ॳظόʔδχΞ෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ ઃఆ͢ΔࡍҙΛʂ
• ύϒϦοΫIP/ElasticIPΛ༩͢Δඞཁ͋Γ (VPNͳͲଓͯ͠Δ߹Λআ͘) • ՝ۚ1࣌ؒ୯Ґ • 1ͬͯɺ59ͬͯಉ͡ྉۚ
EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔͠Εͳ͍͚ͲɺՁ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢՁ֨ͳΒࢢՁ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢՁ֨ >
ೖࡳՁ֨ͱͳΔͱΠϯελϯεऴྃ • 70%off͘Β͍Ͱ͑Δ(ݸਓ࣮) ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ͏ https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html
ͱ͜ΖͰػցֶश͍ͨ͠ͷͰ GPUΛ͍͍ͨ
GPUͬͯͳʹʁ • Graphics Processing Unit • ݩʑɺը૾ॲཧͳͲ͚ʹ։ൃ͞Εͨ • ฒྻԋࢉ͕ಘҙͳϓϩηοα •
ߦྻԋࢉ͕ಘҙ ˠ σΟʔϓϥʔχϯάʹద͍ͯ͠Δ (16 $16
GPU͏ͱͳ͍ͥͷ͔ • ԋࢉग़དྷΔίΞ͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴಡΈࠐΈ
• CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍͍ • σΟʔϓϥʔχϯάͰར༻͢Δߦྻܭࢉ ΛฒྻͰߦ͑ΔͷͰAIج൫Ͱར༻͕ਐΉ
ৄ͘͠GPUͷ͍ͭͯΓ͍ͨํ ͜ͷຊ͕͓͢͢ΊͰ͢ GPUΛࢧ͑Δٕज़ ஶऀɿHisa Ando ൃചɿ20176݄30 URLɿhttps://www.amazon.co.jp/dp/477419056X ※ΞϑΟϦϯΫͰ͋Γ·ͤΜ
AWSͰ͑ΔGPU
GPUΠϯελϯελΠϓ p2Πϯελϯε g2 Πϯελϯε g3Πϯελϯε ←new
ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF
(# /7*%*"5FTMB, (# QYMBSHF (# (# QYMBSHF (# (# HYMBSHF (# /7*%*"(3*%, (# HYMBSHF (# (# HYMBSHF (# /7*%*"5FTMB. (# ౦ژϦʔδϣϯ Ͱ͑ͳ͍ HYMBSHF (# (# HYMBSHF (# (# ※vCPUεϨου
Amazon EMR
Amazon EMRͱ • EC2Πϯελϯε্ʹࢄॲཧج൫Λ ΫϥελʔͰߏங͢ΔαʔϏε • HadoopɺApache SparkڥͳͲ
Amazon EMRͷҙͳͲ • S3ʹ͋ΔΞϓϦέʔγϣϯΛࢦఆ࣮ͯ͠ߦͰ ͖ΔͷͰྑ͍ • ϩάS3ʹग़ྗͰ͖ΔͷͰϩά͕อଘग़དྷΔ • ࢹ͕͍͠ •
ΞϓϦέʔγϣϯଆͰৄࡉͳΤϥʔΛग़ྗͨ͠΄͏͕͍͍ • EMRͰޭ/ࣦഊ͔͠ग़ྗ͞Εͳ͍
Amazon S3
σʔλϨΠΫ • σʔλϨΠΫͱ • ੜϩάɺը૾ɾө૾ɺԻɺͳͲͷ ʮඇߏԽσʔλʯΛཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE
Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet
Amazon S3 • AWSͷετϨʔδαʔϏε σʔλϨΠΫͱͯ͑͠Δ • ಛ • ٱੑ 99.999999999%
• 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμϯλΠϜ 5234ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c
Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४
– ԽετϨʔδ • ॏཁੑͷ͍࠶ੜՄೳͳσʔλ͚ • S3 ඪ४ – සΞΫηε • ΞΫηε͕গͳ͍σʔλ͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/
Amazon S3ͷྉۚମܥ • ετϨʔδྉۚ • ͲΕ͘Β͍σʔλྔΛஔ͍ͯΔ͔ • σʔλసૹྔ • ͲΕ͘Β͍σʔλΛ࣋ͪग़͔ͨ͠
• APIίʔϧྉۚ • Ξοϓϩʔυ/আ/μϯϩʔυͳͲͷૢ࡞
Amazon S3ͷҙ • σʔλసૹྔ͕݁ߏߴ͍Ͱ͢ ex.1TB(1024GB)ΛΠϯλʔωοτܦ༝Ͱ μϯϩʔυͨ͠Β1.5ສԁ($143.36) • σϑΥϧτͷอઌ͕ ถࠃ౦෦(όʔδχΞ)
Amazon Redshift
Amazon Redshiftͱ • AWS͕ఏڙ͢ΔશϚωʔδυܕͷ σʔλΣΞϋε(DHW) • ϖλόΠτΫϥεʹεέʔϧ͢Δ • ྻࢦσʔλϕʔε •
ྉۚମܥΦϯσϚϯυྉۚ
Amazon Redshiftͷҙ • σʔλߏͷઃܭΛ͢Δ(͋ͨΓલ?) • ಛʹࢄΩʔɺιʔτΩʔɺ࣌ܥྻςʔϒϧׂ • ࣌ʑɺڧ੍ϝϯςφϯε͕ൃੜ͢Δ • ϦΞϧλΠϜॲཧʹෆ͖
• ͋͘·Ͱੳ༻ʹ͏ • ຊ൪ڥͰར༻͢Δ߹μϯλΠϜΛఆ͢Δ • μογϡϘʔυ༏ल
հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ͷσʔλʹ
SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ ԾωοτϫʔΫαʔϏε جຊແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷࢹπʔϧ
ΫϥυͰࣄނΛ͙ͨΊʹ
ઈରΔ͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ Ξϥʔτͱ௨ͰٻֹΛϞχλϦϯά http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷཧެ։ઈର͠ͳ͍ • Access key/Secret
Keyඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000ٻɺٽ͖ͦ͏ʹͳ͓ͬͨʯͳͲ • ݟੵΓඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢
࣭λΠϜ
AWSσʔλαΠΤϯε Γ͍ͨͻͱ ϚδͰઈࢍืूத ※ΦϑΟεݟֶ͚ͩͰେৎͰ͢ αΠόʔΤʔδΣϯτ ΞυςΫελδΦ https://adtech.cyberagent.io/
͋Γ͕ͱ͏͍͟͝·ͨ͠
ࢀߟใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS ΫϥυαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ
https://www.youtube.com/user/AmazonWebServices ຊޠ https://www.youtube.com/user/AmazonWebServicesJP AWSΫϥυσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html