データサイエンスのためのAWSの使い方入門

データサイエンスのためのAWSの使い方入門
サポーターズCoLab勉強会 Jul 20 2017
村井 翔太朗

Transcript

1. σʔλαΠΤϯεͷͨΊͷ AWSͷ࢖͍ํೖ໳ αϙʔλʔζCoLabษڧձ Jul 20 2017 ଜҪ ᠳଠ࿕
2. ࣗݾ঺հ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ ※ຊ೔ͷൃද͸ݸਓͷݟղͰ͋Γɺॴଐ͢Δ૊৫ͷެࣜݟղͰ͸͋Γ·ͤΜ
3. ͓·͑AWSৄ͍͠ͷʁ
4. ਓฒΈʹ͸஌ͬͯΔ(͸ͣ) • AWSྺɿ3೥͘Β͍ • AWSͰ޷͖ͳαʔϏεɿRedshiftɺLambda • AWSೝఆɿAWS SAA • Re:Invent 2015 ࢀՃ͠·ͨ͠
5. ΊͪΌͪ͘Όৄ͋͘͠Γ·ͤΜ ઌ೔AWS SAPམͪ·ͨ͠…
6. օ༷ʹ࣭໰͍ͤͯͩ͘͞͞
7. σʔλαΠΤϯςΟετͷํ
8. σʔλ෼ੳ͸͠ͳ͍͚Ͳ
 σʔλ෼ੳج൫ͷߏஙΛ͢Δํ
9. ͳΜͰͦ΋ͦ΋ࠓ೔ͷ࿩͠͸ σʔλαΠΤϯεͱAWSͳͷʁ
10. ༑ਓ ࠷ۙɺσʔλαΠΤϯεςΟετʹͳͬͨΘ ͦΕ͸ͦΕ͸ɺ͓څྉ͗ΐ͏͞Μ΋͏ͯΔΜ΍Ζ ଜҪ ϕϯνϟʔ΍͔Β·ͩ·ͩ͜Ε͔Βɻ ·ͩGPUͱ͔αʔόʔΛങ͏͓ۚ΋ͳ͍͠ɻ Ͱ΋ֶशͤ͞Δͷʹ਺೔͔͔ΔͷΑͶʔ ͋ΕɺΫϥ΢υ࢖ͬͯͳ͍ͷʁ Ϋϥ΢υͬͯ࢖ͬͨ͜ͱͳ͍͚Ͳߴ͍Ͱ͠ΐ ͋ͱ࢖͏ͷ΋೉ͦ͠͏΍͠ ωοτϫʔΫͱ͔ߏஙͱ͔΋Ͱ͖ͳ͍͠ ͦΜͳ͜ͱ೉͍͜͠ͱ͋Β΁ΜͰ ͦ͏ͳͷʁ ΄ͳࠓ౓ίʔώʔ͓͝Δ͔Β͓͑ͯ͠΍ (130ԁͰ͸ͪΐͬͱ…)
11. ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹ෼ੳ༻ͷΫΤϦΛ౤͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗΋͍ͨ͠
 ͬͯݴΘΕͨͷ΍͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜Ε͸ͨͿΜΫϥ΢υҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥ΢υͳͷʁ ن໛ʹ΋ΑΓ·͕͢Ϋϥ΢υͷ΄͏͕
 ։ൃεϐʔυ͕଎͘ɺίετ΋҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜ΍ͬͨΒ
 ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…Զʹ΋λεΫ͕) ͜Ε͚ͩ΍ͱ΍Ε·΁Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞΋Β͍·͢Θɻ
12. ͦΜͳਓ͸ੈͷதʹ ͪΐͬͱ͚͍ͩΔͷͰ͸ͳ͍͔ͱɻ
13. ຊ೔ͷର৅ऀ • Ϋϥ΢υ࢖͍͍ͨσʔλαΠΤϯςΟετͷํ • σʔλαΠΤϯςΟετ(Ά͍ਓ)ʹ
 ج൫ͷߏஙΛ͓ئ͍͞Εͨํ • AWS࢖ͬͨ͜ͱͳ͍ํ • ࣗࣾ/ֶߍͰσʔλ෼ੳج൫Λ࣋ͬͯͳ͍ํ
14. ࠓ೔͓࿩͢Δ͜ͱ • AWSͱ͸ • σʔλαΠΤϯεͱ͸ • AWSαʔϏε঺հ • Amazon EC2 • Amazon EMR • Amazon S3 • Amazon Redshift • Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ
15. AWSͱ͸ʁ
16. AWS(Amazon Web Service) ͱ͸ Ϋϥ΢υίϯϐϡʔςΟϯάαʔϏε →Ϩϯλϧαʔόʔͷ͍͢͝΍ͭ ʮΠϯϑϥ͕WebαʔϏεʹͳͬͨʯ AWS ʹ͍ͭͯ https://aws.amazon.com/jp/about-aws/
17. AWS͸ੈքதʹ͋Δ • Ϧʔδϣϯ • େ཮ͳͲ஍ཧతʹ཭ΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ • ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳ਺ͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿ඼઒ɺཱ઒
 ※࣮ࡍʹ͸ެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
18. AWSͰग़དྷΔ͜ͱ(΄ΜͷҰ෦) • ؆୯ʹߏஙͰ͖Δίϯϐϡʔτ(αʔόʔ) • EC2 • ΫϦοΫ͚ͩͰग़དྷΔσʔλϕʔεͷߏங • RDS • ੍ݶແ͠ͷσʔλετϨʔδ • S3 • ετϦʔϜॲཧ • Kinesis (Stream)
19. σʔλαΠΤϯεͱ͸ʁ σʔλΛऩूɺ෼ੳɺར༻ͯ͠༗ޮ׆༻͢Δٕज़·ͨ͸ֶ໰
20. σʔλαΠΤϯςΟετͱ͸ σʔλαΠΤϯεྗɺ
 σʔλΤϯδχΞϦϯάྗΛϕʔεʹ
 σʔλ͔ΒՁ஋Λ૑ग़͠ɺ
 Ϗδωε՝୊ʹ౴͑Λग़͢
 ϓϩϑΣογϣφϧ
 ※͜͜ͰʮϏδωεʯͱ͸ࣾձʹ໾ʹཱͭҙຯͷ͋Δ׆ಈશൠΛࢦ͢ Ұൠࣾஂ๏ਓσʔλαΠΤϯςΟετڠձ σʔλαΠΤϯςΟετͷϛογϣϯɺεΩϧηοτɺఆٛɺεΩϧϨϕϧΛൃද http://www.datascientist.or.jp/news/2014/pdf/1210.pdf
21. ͳͥσʔλαΠΤϯεͰ AWS(Ϋϥ΢υ)Λ࢖͏ͷ͔
22. σʔλ෼ੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλ΢ΣΞϋ΢εʹ෼ੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ • ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔ΂ͨ͘͘͠ͳ͍ • ߏ੒ͳͲࣗ༝౓͕ߴ͍
23. ඞཁͳ࣌ʹ ඞཁͳϦιʔε͚ͩΛ֬อ͠ ඞཁͳ෼͚ͩ෼ੳ͍ͨ͠ Ϋϥ΢υ͕࠷దͰ͢Αʂ
24. σʔλ෼ੳ͢Δࡍʹ
 AWSͷͲͷαʔϏε͕࢖͑Δ͔
25. ঺հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift Ծ૝αʔόߏஙαʔϏε εέʔϥϒϧͰ ࢖ͬͨ෼͚ͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อ؅Ͱ͖Δ ੩తίϯςϯπͷWebϗεςΟϯά΋Մೳ AWS͕ఏڙ͢Δ ׬શϚωʔδυͷσʔλ΢ΣΞϋ΢ε SQL͕࣮ߦͰ͖ϖλόΠτن໛ͷ σʔλΛѻ͑Δ
26. Amazon EC2
27. Amazon EC2ͱ͸ • Ϋϥ΢υίϯϐϡʔςΟϯά • ͍ΘΏΔԾ૝αʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ • ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛෋ͳΠϯελϯελΠϓ Amazon EC2 ͱ͸ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html
28. Amazon EC2ͷ஫ҙ఺ • ॳظ͸όʔδχΞ๺෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ
 ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ
 ઃఆ͢Δࡍ͸஫ҙΛʂ • ύϒϦοΫIP/ElasticIPΛ෇༩͢Δඞཁ͋Γ
 (VPNͳͲ઀ଓͯ͠Δ৔߹Λআ͘) • ՝ۚ͸1࣌ؒ୯Ґ • 1෼࢖ͬͯ΋ɺ59෼࢖ͬͯ΋ಉ͡ྉۚ
29. EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔΋͠Εͳ͍͚Ͳɺ௿Ձ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢ৔Ձ֨ͳΒࢢ৔Ձ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢ৔Ձ֨ > ೖࡳՁ֨ͱͳΔͱΠϯελϯε͸ऴྃ • 70%off͘Β͍Ͱ࢖͑Δ(ݸਓ࣮੷)
 ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ࢖͏
 https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html
30. ͱ͜ΖͰػցֶश͍ͨ͠ͷͰ GPUΛ࢖͍͍ͨ
31. GPUͬͯͳʹʁ • Graphics Processing Unit • ݩʑɺը૾ॲཧͳͲ޲͚ʹ։ൃ͞Εͨ • ฒྻԋࢉ͕ಘҙͳϓϩηοα • ߦྻԋࢉ͕ಘҙ
 ˠ σΟʔϓϥʔχϯάʹద͍ͯ͠Δ (16 $16
32. GPU࢖͏ͱͳͥ଎͍ͷ͔ • ԋࢉग़དྷΔίΞ਺͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴ଎ಡΈࠐΈ • CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍଎͍ • σΟʔϓϥʔχϯάͰ΋ར༻͢Δߦྻܭࢉ
 Λ௒ฒྻͰߦ͑ΔͷͰAIج൫Ͱ΋ར༻͕ਐΉ
33. ৄ͘͠GPUͷ͍ͭͯ஌Γ͍ͨํ͸ ͜ͷຊ͕͓͢͢ΊͰ͢ GPUΛࢧ͑Δٕज़
 ஶऀɿHisa Ando ൃച೔ɿ2017೥6݄30೔ URLɿhttps://www.amazon.co.jp/dp/477419056X ※ΞϑΟϦϯΫͰ͸͋Γ·ͤΜ
34. AWSͰ࢖͑ΔGPU
35. GPUΠϯελϯελΠϓ p2Πϯελϯε g2 Πϯελϯε g3Πϯελϯε ←new
36. ౥ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF   (# /7*%*"
    5FTMB
    , (# QYMBSHF   (# (# QYMBSHF   (# (# HYMBSHF   (# /7*%*"
    (3*%
    , (# HYMBSHF   (# (# HYMBSHF   (# /7*%*"
    5FTMB
    . (# ౦ژϦʔδϣϯ
    Ͱ͸࢖͑ͳ͍ HYMBSHF   (# (# HYMBSHF   (# (# ※vCPU͸εϨου਺
37. Amazon EMR
38. Amazon EMRͱ͸ • EC2Πϯελϯε্ʹ෼ࢄॲཧج൫Λ
 ΫϥελʔͰߏங͢ΔαʔϏε • HadoopɺApache Spark؀ڥͳͲ
39. Amazon EMRͷ஫ҙ఺ͳͲ • S3ʹ͋ΔΞϓϦέʔγϣϯΛࢦఆ࣮ͯ͠ߦͰ ͖ΔͷͰྑ͍ • ϩά΋S3ʹग़ྗͰ͖ΔͷͰϩά͕อଘग़དྷΔ • ؂ࢹ͕೉͍͠ • ΞϓϦέʔγϣϯଆͰৄࡉͳΤϥʔΛग़ྗͨ͠΄͏͕͍͍ • EMRͰ͸੒ޭ/ࣦഊ͔͠ग़ྗ͞Εͳ͍
40. Amazon S3
41. σʔλϨΠΫ • σʔλϨΠΫͱ͸ • ੜϩάɺը૾ɾө૾ɺԻ੠ɺͳͲͷ
 ʮඇߏ଄ԽσʔλʯΛ؅ཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ
 ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet
42. Amazon S3 • AWSͷετϨʔδαʔϏε
 σʔλϨΠΫͱͯ͠΋࢖͑Δ • ಛ௃ • ଱ٱੑ 99.999999999% • 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ೥͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμ΢ϯλΠϜ 52෼34ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c
43. Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४ – ௿৑௕ԽετϨʔδ • ॏཁੑͷ௿͍࠶ੜՄೳͳσʔλ޲͚ • S3 ඪ४ – ௿ස౓ΞΫηε • ΞΫηε͕গͳ͍σʔλ޲͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/
44. Amazon S3ͷྉۚମܥ • ετϨʔδྉۚ • ͲΕ͘Β͍σʔλྔΛஔ͍ͯΔ͔ • σʔλసૹྔ • ͲΕ͘Β͍σʔλΛ࣋ͪग़͔ͨ͠ • APIίʔϧྉۚ • Ξοϓϩʔυ/࡟আ/μ΢ϯϩʔυͳͲͷૢ࡞
45. Amazon S3ͷ஫ҙ఺ • σʔλసૹྔ͕݁ߏߴ͍Ͱ͢ ex.1TB(1024GB)ΛΠϯλʔωοτܦ༝Ͱ
 μ΢ϯϩʔυͨ͠Β໿1.5ສԁ($143.36) • σϑΥϧτͷอ؅ઌ͕
 ถࠃ౦෦(όʔδχΞ)
46. Amazon Redshift
47. Amazon Redshiftͱ͸ • AWS͕ఏڙ͢Δ׬શϚωʔδυܕͷ
 σʔλ΢ΣΞϋ΢ε(DHW) • ϖλόΠτΫϥεʹεέʔϧ͢Δ • ྻࢦ޲σʔλϕʔε • ྉۚମܥ͸ΦϯσϚϯυྉۚ
48. Amazon Redshiftͷ஫ҙ఺ • σʔλߏ଄ͷઃܭΛ͢Δ(͋ͨΓલ?) • ಛʹ෼ࢄΩʔɺιʔτΩʔɺ࣌ܥྻςʔϒϧ෼ׂ • ࣌ʑɺڧ੍ϝϯςφϯε͕ൃੜ͢Δ • ϦΞϧλΠϜॲཧʹ͸ෆ޲͖ • ͋͘·Ͱ෼ੳ༻ʹ࢖͏ • ຊ൪؀ڥͰར༻͢Δ৔߹͸μ΢ϯλΠϜΛ૝ఆ͢Δ • μογϡϘʔυ͸༏ल
49. ঺հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ಺ͷσʔλʹ SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ Ծ૝ωοτϫʔΫαʔϏε جຊ͸ແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷ؂ࢹπʔϧ
50. Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ
51. ઈର΍Δ΂͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ
 Ξϥʔτͱ௨஌Ͱ੥ٻֹΛϞχλϦϯά
 http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷ؅ཧ͸ެ։ઈର͠ͳ͍ • Access key/Secret Key͸ඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000੥ٻɺٽ͖ͦ͏ʹͳ͓ͬͨ࿩ʯͳͲ • ݟੵ΋Γ͸ඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ
 http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ
 https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ͸[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯౰Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢
52. ࣭໰λΠϜ
53. AWS΍σʔλαΠΤϯε ΍Γ͍ͨͻͱ ϚδͰઈࢍืूத ※ΦϑΟεݟֶ͚ͩͰ΋େৎ෉Ͱ͢ αΠόʔΤʔδΣϯτ ΞυςΫελδΦ https://adtech.cyberagent.io/
54. ͋Γ͕ͱ͏͍͟͝·ͨ͠
55. ࢀߟ৘ใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ https://www.youtube.com/user/AmazonWebServices ೔ຊޠ https://www.youtube.com/user/AmazonWebServicesJP 
AWSΫϥ΢υσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱ෼ੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html