Upgrade to Pro — share decks privately, control downloads, hide ads and more …

1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw awa...

1日100個以上のHadoopクラスターを使い捨てる方法 / How to throw away 100 Hadoop clusters a day

More Decks by Hiroyuki Nakazato / 中里 浩之

Other Decks in Technology

Transcript

  1.   ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT

    &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ
  2.   ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT

    &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ εέʔϥϏϦςΟ ͳͲʹ՝୊
  3.   ϩάऩू γεςϜ 4 3FETIJGU ,JOFTJT +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT

    &-# &$ ऩूɾ஝ੵ &5- ղੳ &$ &$ ':ΞʔΩςΫνϟʔ ՝୊ͷղܾʹணख
  4.   ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5-

    ղੳ ऩूɾ஝ੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ৽ΞʔΩςΫνϟʔ (MVF
  5.   &5-ॲཧͷεςοϓ &.3 ຊ൪ࢿࡐ +BS 4IFMM ςετࢿࡐ 4IFMM +BS

    ੜϩά 1BSRVFU  &5-ॲཧ༻ͷࢿࡐΛ4͔Βμ΢ϯϩʔυ  TEJTUDQͰॲཧର৅ͷϩάΛू໿  TQBSLTVCNJUͰσʔλՃ޻ͱ1BSRVFUม׵Λ࣮ࢪ 1 2 3
  6.   -BNCEBʹΑΔΫϥελʔ੍ޚ 1 2 3 4 &.3Ϋϥελʔىಈ༻ͷ-BNCEBؔ਺Λ࣮૷ 1ZUIPOͰ3VO+PC'MPXΛίʔϧ 3VO+PC'MPX͸ύϥϝʔλʔ͕ඇৗʹଟ͍

    ઃఆϑΝΠϧϥΠϒϥϦ)0$0/Ͱ؅ཧ͠࠶ར༻ੑΛ޲্ 1MBZ'SBNFXPSLͳͲͰ༗໊ͳ-JHIUCFOE͕ࣾ։ൃ ϩάऩूγεςϜͳͲͰ1MBZ'SBNFXPSLΛ׆༻ͯ͠ ͓Γ)0$0/ʹ׳Ε͍ͯͨͷ͕࠾༻ཧ༝
  7.   &5-༻Ϋϥελʔىಈϑϩʔ 1 2 3 4 Name = hourly-etl

    ReleaseLabel = emr-5.20.0 Steps = [ { spark-submit hourly-etl.jar ${YYYYMMDD}/${HH} } ... IPVSMZDPOG  IPVSMZDPOG )0$0/ύʔε 3VO+PC'MPX࣮ߦ
  8.   &5-ॲཧͷϦΧόϦʔ ੒ޭ ੒ޭ ࣦഊ ͕ҟৗऴྃ 1 2 3

    4 ϦΧόϦ༻ ىಈ༻ Ϋϥελʔͷঢ়ଶΛ ఆظతʹνΣοΫ ىಈ༻-BNCEBΛ ݺͼग़͠ ࣦഊͨ࣌ؒ͠ͷ ΫϥελʔΛىಈ
  9.   ϑϧϚωʔδυ)JWFϝλετΞ 1 2 3 4 "84(MVF%BUB$BUBMPH <&.3 ରԠ>

    Ҏલ͸.Z42-PO3%4Λ)JWFϝλετΞͱͯ͠ར༻ 3%4͕410'ʹͳΒͳ͍Α͏ʹؾΛݣ͏ඞཁ͕͋ͬͨ ΫϥελʔΛେྔʹىಈ͢Δͱ઀ଓͰ͖ͳ͍FUD (MVF%BUB$BUBMPH͸ϑϧϚωʔδυ͔ͭಉ࣌઀ଓ਺੍ݶͳ͠
  10.   Ϋϥελʔͷམͱ͠๨Ε๷ࢭ 1 2 3 4 I I I

    ؂ࢹ༻ Ϋϥελʔͷ Քಇ࣌ؒΛνΣοΫ 4MBDL௨஌ BEIPDDMVTUFS 
  11.   ϩάऩू γεςϜ 4 3FETIJGU +FOLJOTPO&$ %FWFMPQFST %BUB"OBMZTUT &5-

    ղੳ ऩूɾ஝ੵ ʜ ʜ 4QBSLPO&.3 )JWFPO&.3 ʜ 4 -BNCEB ,JOFTJT &-# &$ ࠶ܝ৽ΞʔΩςΫνϟʔ (MVF