Upgrade to Pro — share decks privately, control downloads, hide ads and more …

データサイエンスのためのAWSの使い方入門

 データサイエンスのためのAWSの使い方入門

データサイエンスのためのAWSの使い方入門
サポーターズCoLab勉強会 Jul 20 2017
村井 翔太朗

Show Murai

July 21, 2017
Tweet

More Decks by Show Murai

Other Decks in Technology

Transcript

  1. ࣗݾ঺հ • ଜҪᠳଠ࿕ • גࣜձࣾαΠόʔΤʔδΣϯτ • ΞυςΫຊ෦ ΠϯϑϥνʔϜ • ΠϯϑϥΤϯδχΞ

    ※ຊ೔ͷൃද͸ݸਓͷݟղͰ͋Γɺॴଐ͢Δ૊৫ͷެࣜݟղͰ͸͋Γ·ͤΜ
  2. ಉ྅ ػցֶश༻ͷαʔόʔͱ σʔλʹ෼ੳ༻ͷΫΤϦΛ౤͛ΔDBͱ ଞ෦ॺͱσʔλͷڞ༗΋͍ͨ͠
 ͬͯݴΘΕͨͷ΍͚Ͳ૬ஊͤͯ͞ʔ ͓ɺ·͍Ͳʂ ͜Ε͸ͨͿΜΫϥ΢υҊ݅Ͱ͢Θ ଜҪ ͑ɺࣗࣾαʔόʔ͡Όͳ͘Ϋϥ΢υͳͷʁ ن໛ʹ΋ΑΓ·͕͢Ϋϥ΢υͷ΄͏͕


    ։ൃεϐʔυ͕଎͘ɺίετ΋҆͘ɺ ӡ༻ָ͕Ͱ͢ͶΜ ΄͏΄͏ɺ΄ͳΘ͔ͬͯΔΜ΍ͬͨΒ
 ͜ͷ͓ئ͍͍͍ͯ͠ʁ (͑…Զʹ΋λεΫ͕) ͜Ε͚ͩ΍ͱ΍Ε·΁Μɻ ҰճͪΌΜͱώΞϦϯάͤͯ͞΋Β͍·͢Θɻ
  3. ࠓ೔͓࿩͢Δ͜ͱ • AWSͱ͸ • σʔλαΠΤϯεͱ͸ • AWSαʔϏε঺հ • Amazon EC2

    • Amazon EMR • Amazon S3 • Amazon Redshift • Ϋϥ΢υͰࣄނΛ๷͙ͨΊʹ
  4. AWS͸ੈքதʹ͋Δ • Ϧʔδϣϯ • େ཮ͳͲ஍ཧతʹ཭ΕͨྖҬ • 14ͭͷϦʔδϣϯ • ౦ژɺόʔδχΞɺϩϯυϯ •

    ΞϕΠϥϏϦςΟʔκʔϯ • 1 ͭͷϦʔδϣϯʹෳ਺ͷͦΕͧΕಠཱͨ͠ϩέʔγϣϯ • ྫ. ౦ژϦʔδϣϯͷAZɿ඼઒ɺཱ઒
 ※࣮ࡍʹ͸ެ։͞Ε͍ͯ·ͤΜ EC2 ϦʔδϣϯͱΞϕΠϥϏϦςΟʔκʔϯ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-regions-availability-zones.html
  5. σʔλ෼ੳʹٻΊΒΕΔ͜ͱ • σʔλΞφϦςΟΫε • େ͖͍σʔλΛ҆Ձʹѻ͍͍ͨ • σʔλ΢ΣΞϋ΢εʹ෼ੳΫΤϦΛ࣮ߦ͍ͨ͠ • ؆୯ʹՄࢹԽ͍ͨ͠ •

    ػցֶश࣮ߦج൫ • ֶश࣌”͚ͩ”ඞཁʹͳΔେྔͷGPU • ӡ༻ΛͳΔ΂ͨ͘͘͠ͳ͍ • ߏ੒ͳͲࣗ༝౓͕ߴ͍
  6. ঺հ͢ΔαʔϏε Amazon EC2 Amazon S3 Amazon EMR Amazon Redshift Ծ૝αʔόߏஙαʔϏε

    εέʔϥϒϧͰ ࢖ͬͨ෼͚ͩͷैྔ՝ۚ ϏοάσʔλϑϨʔϜϫʔΫΛ ؆୯ʹߏஙͰ͖ΔϚωʔδυαʔϏε HadoopɺSparkΫϥελʔ͕؆୯ʹߏஙͰ͖Δ σʔλετϨʔδαʔϏε ແ੍ݶʹσʔλ͕อ؅Ͱ͖Δ ੩తίϯςϯπͷWebϗεςΟϯά΋Մೳ AWS͕ఏڙ͢Δ ׬શϚωʔδυͷσʔλ΢ΣΞϋ΢ε SQL͕࣮ߦͰ͖ϖλόΠτن໛ͷ σʔλΛѻ͑Δ
  7. Amazon EC2ͱ͸ • Ϋϥ΢υίϯϐϡʔςΟϯά • ͍ΘΏΔԾ૝αʔόͷߏங • ྉۚମܥ • ίϯϐϡʔτͷىಈ࣌ؒʹԠͯ͡՝ۚ

    • ϘϦϡʔϜʹର͢Δ՝ۚ • ωοτϫʔΫసૹྔʹର͢Δ՝ۚ • ๛෋ͳΠϯελϯελΠϓ Amazon EC2 ͱ͸ http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/concepts.html
  8. Amazon EC2ͷ஫ҙ఺ • ॳظ͸όʔδχΞ๺෦ϦʔδϣϯͰىಈ • ηΩϡϦςΟάϧʔϓͰϙʔτΛղ์͠ͳ͍ͱ
 ωοτϫʔΫͷૄ௨͕ग़དྷͳ͍ • ηΩϡϦςΟάϧʔϓͰશެ։[0.0.0.0/0]Λ
 ઃఆ͢Δࡍ͸஫ҙΛʂ

    • ύϒϦοΫIP/ElasticIPΛ෇༩͢Δඞཁ͋Γ
 (VPNͳͲ઀ଓͯ͠Δ৔߹Λআ͘) • ՝ۚ͸1࣌ؒ୯Ґ • 1෼࢖ͬͯ΋ɺ59෼࢖ͬͯ΋ಉ͡ྉۚ
  9. EC2εϙοτΠϯελϯε • Amazonʹམͱ͞ΕΔ͔΋͠Εͳ͍͚Ͳɺ௿Ձ֨ͳΠϯελϯε • ೖࡳՁ֨ > ࢢ৔Ձ֨ͳΒࢢ৔Ձ֨ͰΠϯελϯεΛىಈͰ͖Δ • ࢢ৔Ձ֨ >

    ೖࡳՁ֨ͱͳΔͱΠϯελϯε͸ऴྃ • 70%off͘Β͍Ͱ࢖͑Δ(ݸਓ࣮੷)
 ※ΦϯσϚϯυΠϯελϯεͱͷൺֱ • ೖࡳΞυόΠβʔΛ࢖͏
 https://aws.amazon.com/jp/ec2/spot/bid-advisor/ Amazon EC2 εϙοτΠϯελϯε http://docs.aws.amazon.com/ja_jp/AWSEC2/latest/UserGuide/using-spot-instances.html
  10. GPU࢖͏ͱͳͥ଎͍ͷ͔ • ԋࢉग़དྷΔίΞ਺͕2000-4000ίΞ • ฒྻܭࢉॲཧʹ༗ར • CPU(Intel Xeonͱ͔)ͩͱ࠷େ48ίΞ͘Β͍ • ϝϞϦ͕ಠཱ͔ͭߴ଎ಡΈࠐΈ

    • CPUͷϝϞϦόϯυ෯ΑΓ10ഒ͘Β͍଎͍ • σΟʔϓϥʔχϯάͰ΋ར༻͢Δߦྻܭࢉ
 Λ௒ฒྻͰߦ͑ΔͷͰAIج൫Ͱ΋ར༻͕ਐΉ
  11. ౥ࡌGPUΠϯελϯε Πϯελϯε $16 W$16 ϝϞϦ (16 (16ϝϞϦ උߟ QYMBSHF 

     (# /7*%*"5FTMB, (# QYMBSHF   (# (# QYMBSHF   (# (# HYMBSHF   (# /7*%*"(3*%, (# HYMBSHF   (# (# HYMBSHF   (# /7*%*"5FTMB. (# ౦ژϦʔδϣϯ Ͱ͸࢖͑ͳ͍ HYMBSHF   (# (# HYMBSHF   (# (# ※vCPU͸εϨου਺
  12. σʔλϨΠΫ • σʔλϨΠΫͱ͸ • ੜϩάɺը૾ɾө૾ɺԻ੠ɺͳͲͷ
 ʮඇߏ଄ԽσʔλʯΛ؅ཧ͢ΔྖҬ • ඞཁͳ࣌ʹσʔλϨΠΫ͔ΒऔΓग़͠ɺ
 ͙͢͞·σʔλΛ׆༻Ͱ͖Δ GE

    Announces First Data Lake Approach for Industrial Internet to Better Access, Analyze and Store Industrial-Strength Big Data http://www.businesswire.com/news/home/20140810005024/en/GE-Announces-Data-Lake-Approach-Industrial-Internet
  13. Amazon S3 • AWSͷετϨʔδαʔϏε
 σʔλϨΠΫͱͯ͠΋࢖͑Δ • ಛ௃ • ଱ٱੑ 99.999999999%

    • 1ສݸͷΦϒδΣΫτͷ͏ͪ1ͭͷফࣦʹ1000ສ೥͔͔Δ • Մ༻ੑ 99.99% • S3αʔϏεμ΢ϯλΠϜ 52෼34ඵ • ࡉ͔͘ઃఆͰ͖ΔΞΫηείϯτϩʔϧ • ACLɺόέοτϙϦγʔɺAIMϙϦγʔ S3ͷΞΫηείϯτϩʔϧ·ͱΊ http://qiita.com/ryo0301/items/791c0a666feeea0a704c
  14. Amazon S3ͷछྨ • S3 ඪ४ • ී௨ͷετϨʔδ • S3 ඪ४

    – ௿৑௕ԽετϨʔδ • ॏཁੑͷ௿͍࠶ੜՄೳͳσʔλ޲͚ • S3 ඪ४ – ௿ස౓ΞΫηε • ΞΫηε͕গͳ͍σʔλ޲͚ • ΞʔΧΠϒ(Amazon Glacier) • ΊͬͨʹΞΫηε͞Εͳ͍σʔλ༻ https://aws.amazon.com/jp/s3/storage-classes/
  15. ঺հ͔ͨͬͨ͠αʔϏε Amazon RDS AWS͕ఏڙ͢ΔϚωʔδυܕ ϦϨʔγϣφϧσʔλϕʔε Amazon Athena Amazon S3 ಺ͷσʔλʹ

    SQL Λ࣮ߦͰ͖ΔαʔϏε 6݄ʹ౦ژϦʔδϣϯ։࢝ Amazon Kinesis Amazon Elasticsearch AWS Lambda Amazon VPC Amazon CloudWatch Amazonͷఏڙ͢Δ Ծ૝ωοτϫʔΫαʔϏε جຊ͸ແྉͰར༻Մೳ ετϦʔϛϯάॲཧ͕ ग़དྷΔϚωʔδυαʔϏε αʔόʔϨεͰϓϩάϥϜΛ ࣮ߦͰ͖ΔαʔϏε ElasticsearchΛΤϯδϯ ͱͨ͠ϚωʔδυαʔϏε AWSͷ؂ࢹπʔϧ
  16. ઈର΍Δ΂͖͜ͱ • ྉۚΞϥʔϜΛઃఆ͢Δ
 Ξϥʔτͱ௨஌Ͱ੥ٻֹΛϞχλϦϯά
 http://docs.aws.amazon.com/ja_jp/awsaccountbilling/latest/aboutv2/monitor-charges.html • APIΩʔͷ؅ཧ͸ެ։ઈର͠ͳ͍ • Access key/Secret

    Key͸ඞͣެ։͠ͳ͍Α͏ʹ • ʮॳ৺ऀ͕AWSͰϛεͬͯෆਖ਼ར༻͞Εͯ$6,000੥ٻɺٽ͖ͦ͏ʹͳ͓ͬͨ࿩ʯͳͲ • ݟੵ΋Γ͸ඞ͓ͣ͜ͳ͏ • AWS ؆қݟੵΓπʔϧ
 http://calculator.s3.amazonaws.com/index.html?lng=ja_JP • AWS ՝ۚମܥͱݟੵΓํ๏ʹ͍ͭͯ
 https://aws.amazon.com/jp/how-to-understand-pricing/ • ηΩϡϦςΟάϧʔϓ͸[ssh 0.0.0.0/0]ͱ͔ઃఆ͠ͳ͍ • ݁ߏϒϧʔτϑΥʔε(૯౰Γ߈ܸ)ͱ͔ɺ੬ऑੑ߈ܸ͕͋Γ·͢
  17. ࢀߟ৘ใ AWS υΩϡϝϯτ https://aws.amazon.com/jp/documentation/ AWS Ϋϥ΢υαʔϏε׆༻ࢿྉू https://aws.amazon.com/jp/aws-jp-introduction/ Youtube AWSެࣜνϟϯωϧ(Re:InventͷηογϣϯͳͲ) ӳޠ

    https://www.youtube.com/user/AmazonWebServices ೔ຊޠ https://www.youtube.com/user/AmazonWebServicesJP 
AWSΫϥ΢υσβΠϯύλʔϯ http://aws.clouddesignpattern.org/index.phpύλʔϯ Developers.IO http://dev.classmethod.jp/ DeNA TechCon 2017 ͱ Developers Summit 2017 ͰDeNAͷػցֶशج൫ͱ෼ੳج൫ͷߨԋΛ͠·ͨ͠ http://blog.livedoor.jp/sonots/archives/49502478.html