Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DMMでのPrestoおよび周辺エコシステムの運用事例

DMM.com
June 16, 2017

 DMMでのPrestoおよび周辺エコシステムの運用事例

Presto Meetup 201706にて、発表した弊社のPresto運用事例紹介です。
https://techplay.jp/event/621143

DMM.com

June 16, 2017
Tweet

More Decks by DMM.com

Other Decks in Technology

Transcript

  1. 2 Copyright © since 1998 DMM All Rights Reserved. ࣗݾ঺հ

    Ϗοάσʔλج൫ͷӡ༻อक 1SFTUPྺɿ೥Ґ (P +BWB 3VCZ 1)1 ౻Ҫ྄ଠ %..DPNϥϘγεςϜຊ෦Ϗοάσʔλ෦
  2. 4 Copyright © since 1998 DMM All Rights Reserved. αʔϏεʹ͍ͭͯ

    %..DPN͸೥͔Βଓ͘࿝ฮ αʔϏεαΠτͰ͢ɻ࣌୅ͷχʔζ ʹ߹Θͤͨଟ࠼ͳίϯςϯπΛɺ ೥ؒഓͬͨಠࣗϓϥοτϑΥʔϜͰ ҆ఆతʹఏڙ͍ͯ͠·͢ɻ "CPVU%..DPN Ҏ্ͷ෯޿͍αʔϏεΛల։
  3. 6 Copyright © since 1998 DMM All Rights Reserved. ΞδΣϯμ

    w γεςϜߏ੒ w ֤ίϯϙʔωϯτͷ໾ׂ w 1SFTUPϦιʔεͷ׆༻ w ·ͱΊ
  4. 8 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤ

    1SFTUP ;FQQFMJO 1SFTUP 3FEBTI &MBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ !
  5. 9 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤBOBMZUJDT

    1SFTUP ;FQQFMJO 1SFTUP 3FEBTI &MBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w ΞφϦετ͕೚ҙͷΫΤϦΛ ࣮ߦ͢Δ؀ڥ w 1SFTUPΫΤϦ͸SFEBTI [FQQFMJO͔Β࣮ߦ͢Δ w ௕ظՔಇ͍ͯ͠ΔΫΤϦ͸ɺ ΤϯδχΞ͕؂ࢹ͠ɺঢ়گ ʹԠ͡Ωϟϯηϧ͢Δ ෼ੳ؀ڥ
  6. 10 Copyright © since 1998 DMM All Rights Reserved. γεςϜશମਤCBUDI

    1SFTUP [FQQFMJO 1SFTUP SFEBTI &MBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ ! w γεςϜ͔Βൃߦ͞ΕΔΫΤ Ϧͷ࣮ߦ؀ڥ w IBEPPQ্ͷσʔλΛBQJܦ ༝ͰଞγεςϜʹ࿈ܞ͢Δ ͨΊͷநग़ॲཧΛߦ͏
 IBEPPQʹର͢Δ&5-͸ߦ Θͳ͍  w ఆܕతͳΫΤϦ͕ϝΠϯ όον؀ڥ
  7. 12 Copyright © since 1998 DMM All Rights Reserved. ;FQQFMJO

    1SFTUP 3FEBTI FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! )BEPPQ %..άϧʔϓશମͷϏοάσʔλج൫ $%) 5#1# ECTUBCMFT NPTUMZTUPSFEBT03$TOBQQZ
  8. 13 Copyright © since 1998 DMM All Rights Reserved. 1SFTUPBOBMZUJDTQSFTUP

    ෼ੳ༻Ϋϥελ 1SFTUP DPPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  9. 14 Copyright © since 1998 DMM All Rights Reserved. MPHHJOH

    RVFSZMPHPO 1SFTUP 1SFTUP ΞυϗοΫΫΤϦΛՄࢹԽ͍ͨ͠ IUUQTHJUIVCDPN[[QSFTUPBVEJU
  10. 15 Copyright © since 1998 DMM All Rights Reserved. 3FEBTI#*UPPM

    #*πʔϧ 3FEBTI C JOUFHSBUFHPPHMFPBVUI ;FQQFMJO 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! 3FEBTI
  11. 16 Copyright © since 1998 DMM All Rights Reserved. JOUSPEVDF

    VTFSUP 1SFTUP 3FEBTI QBUDIΛ౰ͯͯΫΤϦͷ࣮ߦϢʔβʔ൑ผ IUUQTHJUIVCDPNHFUSFEBTISFEBTIJTTVFT
  12. 17 Copyright © since 1998 DMM All Rights Reserved. "QBDIF;FQQFMJOOPUFCPPL

    ΞφϦετ༻OPUFCPPL 7FSTJPO4/"14)05 VTF1SFTUPWJBKECD OPUTQBSL  OPUZFUJOUFHSBUFHPPHMFPBVUI 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  13. 18 Copyright © since 1998 DMM All Rights Reserved. JOUSPEVDF

    VTFSUP QSFTUP "QBDIF;FQQFMJO ;FQQFMJOͰར༻͍ͯ͠ΔBQBDIFTIJSPͱQBDKͷ૊Έ߹ ΘͤʹΑΓɺ(PPHMF0"VUIΛར༻Մೳ
  14. 19 Copyright © since 1998 DMM All Rights Reserved. &',FMBTUJDTFBSDI

    qVFOUE LJCBOB ෼ੳΫΤϦͷՄࢹԽ ར༻ঢ়گͷϞχλϦϯά -VDFOF2VFSZʹΑΓɺҟৗΫΤϦΛݕࡧ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  15. 20 Copyright © since 1998 DMM All Rights Reserved. NPOJUPS

     TUBUJTUJDT &', QSFTUPͷར༻ঢ়گΛϞχλϦϯά
  16. 21 Copyright © since 1998 DMM All Rights Reserved. 1SFTUPCBUDIQSFTUP

    όον༻Ϋϥελ 1SFTUP DPPSE XPSLFS ʙRVFSZEBZ OFFETIJHIBWBJMBCJMJUZ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  17. 22 Copyright © since 1998 DMM All Rights Reserved. NVMUJ

    DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
  18. 23 Copyright © since 1998 DMM All Rights Reserved. NVMUJ

    DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
  19. 24 Copyright © since 1998 DMM All Rights Reserved. NVMUJ

    DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
  20. 25 Copyright © since 1998 DMM All Rights Reserved. NVMUJ

    DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ#
  21. 26 Copyright © since 1998 DMM All Rights Reserved. NVMUJ

    DPPSEJOBUPS QSFTUP )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ coordinator=true discovery-server.enabled=true discovery.uri=http://localhost:8080 coordinator=false discovery-server.enabled=false discovery.uri=http://haproxyB:8080 frontend presto bind *:8080 default_backend presto backend presto balance roundrobin option httpchk GET /v1/info/state http-check expect string ACTIVE server active coordB1:8080 check server standby coordB2:8080 check backup DMJFOU IBQSPYZ# DPPSEJOBUPSͷϑΣΠϧΦʔόʔ΋ରԠՄೳ "
  22. 27 Copyright © since 1998 DMM All Rights Reserved. CBUDI

    ֤छσʔλఏڙͷͨΊͷόονॲཧ $SFBUFUBCMFBTTFMFDUͰNBSJBECʹσʔλసૹ 1SFTUPDPPSEJOBUPSͷ੾Γସ͑Λߟྀ͠ɺϦτϥΠΛ࣮૷ 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  23. 28 Copyright © since 1998 DMM All Rights Reserved. $5"4CZ

    XJUICMPDL CBUDI with test as ( select 1 num ) create table hive.adhoc.ctas_test_ng as select * from test ; create table hive.adhoc.ctas_test_ok as with test as ( select 1 num ) select * from test ; XJUI۟ΛDSFBUFจͰ࢖͏৔߹͸ɺઌʹDSFBUFΛॻ͘ /( 0, with test as ( select 2 num ) insert into hive.adhoc.ctas_test_ok select * from test ; insert into hive.adhoc.ctas_test_ok with test as ( select 2 num ) select * from test ;
  24. 29 Copyright © since 1998 DMM All Rights Reserved. ͦͷଞ

    TMBDL 1SFTUPͰTMPXRVFSZͷݕ஌͓ΑͼDBODFM TZTUFNSVOUJNFRVFSJFTΛࢀর͠ɺ͖͍͠஋Ҏ্ͷ௕࣌ؒ Քಇ͍ͯ͠ΔΫΤϦΛݕ஌ˠ#TMBDL ର৅ΫΤϦΛ֬ೝ͠ɺDBODFM 1SFTUP FMBTUJDTFBSDI LJCBOB )BEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! ;FQQFMJO 3FEBTI
  25. 30 Copyright © since 1998 DMM All Rights Reserved. DIFDLLJMM

    TMPXRVFSZ TMBDLXFC)VCPU ࣌ؒͷ͔͔͍ͬͯΔฅΫΤϦͷݕ஌ఀࢭ
  26. 32 Copyright © since 1998 DMM All Rights Reserved. ֤1SFTUPΫϥελͷར༻ঢ়گ

    ෼ੳ1SFTUPΫϥελ DPPSE XPSLFS ΄΅೔தͷΈར༻ όον1SFTUPΫϥελ DPPSE XPSLFS ओʹ໷ؒʹར༻ ໷ؒଳ͸෼ੳ1SFTUPΫϥελΛόον༻్ʹస༻͍ͨ͠
  27. 33 Copyright © since 1998 DMM All Rights Reserved. TXJUDI

    DPPSEJOBUPS CZ)"1SPYZ ೔தଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" ෼ੳ όον
  28. 34 Copyright © since 1998 DMM All Rights Reserved. TXJUDI

    DPPSEJOBUPS CZ)"1SPYZ ໷ؒଳ )"1SPYZ# DPPSE# DPPSE# XPSLFS# XPSLFS# ʜʜʜ DMJFOU IBQSPYZ# )"1SPYZ" DPPSE" XPSLFS" XPSLFS" ʜʜʜ DMJFOU IBQSPYZ" )"1SPYZ$)"1SPYZʹϓϩΩγͯ͠
 ෼ੳ1SFTUPͷXPSLFSΛόονʹͿΒԼ͛ ෼ੳ όον
  29. 35 Copyright © since 1998 DMM All Rights Reserved. TXJUDI

    DPPSEJOBUPS CZ)"1SPYZ ˞஫ҙ఺˞ OPEFFOWJSPONFOU͕ҧ͏XPSLFS͸ೝࣝ͞Εͳ͍ ˠQSPQFSUJFTΛมߋ͠ɺ࠶ىಈ͕ඞཁ # node.properties node.environment=ANALYTICS_PROD node.id=xxx node.data-dir=/var/presto/data # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data ෼ੳ # node.properties node.environment=BATCH_PROD node.id=xxx node.data-dir=/var/presto/data όον
  30. 37 Copyright © since 1998 DMM All Rights Reserved. ·ͱΊ

    #*πʔϧͱ1SFTUPΛ࿈ܞ 1SFTUPͷར༻ঢ়گΛ೺Ѳ͠ɺΤϯδχΞ͕ద੾ʹରॲՄೳ )"1SPYZʹΑΓ1SFTUPDPPSEJOBUPSͷ410'Λআڈ CBUDIͳͲγεςϜ༻్ͷΫϥελͷՄ༻ੑ޲্ ֤Ϋϥελͷར༻ঢ়گʹԠ͡XPSLFSͷৼΓ෼͚ Ϧιʔεͷޮ཰తͳར༻
  31. 39 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO

    1SFTUP SFEBTI FMBTUJDTFBSDI LJCBOB IBEPPQ CBUDI NBSJBEC BQJ 1SFTUP ! QSFTUPBOBMZUJDTQSFTUP ΞυϗοΫΫΤϦ༻Ϋϥελ DPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE
  32. 40 Copyright © since 1998 DMM All Rights Reserved. ֤QSFTUPΫϥελͷར༻ঢ়گ

    ෼ੳ1SFTUPΫϥελ DPPSE XPSLFS ΄΅೔தͷΈར༻ όον1SFTUPΫϥελ DPPSE XPSLFS ओʹ໷ؒʹར༻ ໷ؒଳ͸෼ੳ1SFTUPΫϥελΛόον༻్ʹస༻͍ͨ͠
  33. 41 Copyright © since 1998 DMM All Rights Reserved. [FQQFMJO

    QSFTUP SFEBTI FMBTUJDTFBSDI LJCBOB IBEPPQ CBUDI NBSJBEC BQJ QSFTUP ! QSFTUPBOBMZUJDTQSFTUP ΞυϗοΫΫΤϦ༻Ϋϥελ DPSE XPSLFS RVFSZEBZCZBOBMZTUFOHJOFFS FWFSZNJOGPSDIFDLTMPXRVFSZ TFOERVFSZMPHCZqVFOUE ࡏ੶ΞφϦετɿਓʘ ?P? ʗ ʺΤϯδχΞ