Share
Embed
Transcript
28 slides
-
Copyright © ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͱ3VCZͰ࣮ݱ͢ΔϫʔΫϑϩʔཧ ;0;0508/ͷ
όονσʔλసૹج൫հ גࣜձࣾ;0;0ςΫϊϩδʔζ ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ Ԙ࡚݈߂ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ࣗݾհ ৽ଔͰ7"4*-:ʹೖࣾޙɺ."Λܦ
ͯ;0;0ςΫϊϩδʔζɻੲ 3BJMTͰXFC "1*Λ࡞ͬͨΓɺݕࡧ αʔόʔͷύϑΥʔϚϯεɾνϡʔ χϯάΛͬͨΓɻ࠷ۙͷ͓ࣄ σʔλج൫ͷཧͱ͔ɻ ;0;0ςΫϊϩδʔζ ϚʔέςΟϯάΦʔτϝʔγϣϯνʔϜ Ԙ࡚݈߂ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. ͓͜ͱΘΓ 3BJMTͷ͠·ͤΜʂ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. ࣍ αʔϏεհ σʔλج൫ͷհ σʔλϑϩʔ Πϯϑϥ Ͳ͜Ͱ3VCZΛ͔͍ͭͬͯΔͷ͔ σʔλϚʔτͷࠩߋ৽ͷϩδοΫ STQFDͰ42-ϑΝΠϧͷςετ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ Ҏ্ͷγϣοϓɺ
Ҏ্ ͷϒϥϯυͷऔΓѻ͍ʢ݄ ࣌ʣ ˓ৗ࣌ສҎ্ͷΞΠςϜͱ ຖฏۉ Ҏ্ͷ৽ணΛܝࡌ ˓ଈૹαʔϏεΪϑτϥοϐϯ άαʔϏεπέ͍ͳͲ IUUQ[P[PKQ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ ສμϯϩʔυಥഁɺίʔσΟ
ωʔτߘ૯ສ݅ʢ ݄࣌ʣ ˓શੈքʢ"QQ4UPSF(PPHMFQMBZ ͕ར༻ՄೳͳશͯͷࠃʣͰμϯϩʔυ ͕Մೳ ˓ສਓҎ্ͷϑΥϩϫʔΛ࣋ͭϢʔ βʔʢ8&"3*45"ʣੜ IUUQTXFBSKQ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓͕ࣾಠࣗʹ։ൃͨ͠࠾ੇ༻ϘσΟʔ εʔπ
˓શମʹࢪ͞ΕͨυοτϚʔΧʔΛε ϚʔτϑΥϯΧϝϥͰࡱӨ͢Δ͜ ͱͰɺମܕσʔλΛܭଌ ˓ମܕσʔλɺॠ࣌ʹ%ϞσϧԽ͞ Εɺ;0;0508/ΞϓϦʹอଘɻ IUUQ[P[PKQ[P[PTVJU -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. ˓ʮ;0;046*5ʯͰܭଌͨ͠ମܕσʔ λΛͱʹɺҰਓͻͱΓͷମܕʹ߹ͬ
ͨʮ͋ͳͨαΠζʯͷΞΠςϜ ˓ʮڀۃͷϑΟοτײʯΛ࣮ݱͨ͠ϕʔ γοΫΞΠςϜΛఏڙ ˓άϩʔόϧαΠτʮ;0;0DPNʯ Ͱւ֎ల։ IUUQ[P[PKQQC -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ ;0;0508/%# 8&"3%# 1#%# 1PXFS#* 4 #JH2VFSZ όονసૹ ($4 "OBMZTU σʔλूܭ σʔλ׆༻ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ ;0;0508/%# 8&"3%# 1#%# 1PXFS#* 4 #JH2VFSZ όονసૹ ($4 "OBMZTU σʔλूܭ σʔλ׆༻ ࠓճ͞ͳ͍෦ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. ཁૉٕज़ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. &NCVMLͱ w&5- &YUSBDU5SBOTGFS-PBE πʔϧ w5SFBTVSF%BUBͷ044 wେྔͷσʔλΛόονసૹ wσʔλͷೖྗɾՃɾग़ྗΛ͢Δ෦ΛϓϥάΠϯͱͯ͠ఏڙ wઃఆϑΝΠϧ:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯཧ͕༰қ wϩΰͷγϟν͕ՄѪ͍ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͱ wϫʔΫϑϩʔཧπʔϧ w5SFBTVSF%BUBͷ044 wେྔͷ&NCVMLδϣϒΛޮతʹཧ wฒྻ࣮ߦ wϦτϥΠ wλεΫͷ࣮ߦঢ়گͷՄࢹԽ w࣮ߦϩάͷදࣔ wઃఆϑΝΠϧ΄΅:".-Ͱॻ͔Ε͍ͯΔͷͰόʔδϣϯཧ͕༰қ wϩΰͷϞάϥ͕ՄѪ͍ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ ;0;0508/%# 8&"3%# 1#%# 4 #JH2VFSZ &NCVMLʹΑΔ όονసૹ ($4 σʔλूܭ ϫʔΫϑϩʔཧ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$1FFSJOH %JHEBHNBTUFS "VSPSB %JHEBHXPSLFS "VUP4DBMJOH ɾɾɾ 4 0UIFS71$ ($4 #JH2VFSZ "-# -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$1FFSJOH %JHEBHNBTUFS "VSPSB %JHEBHXPSLFS "VUP4DBMJOH ɾɾɾ 4 0UIFS71$ ($4 #JH2VFSZ "-# δϣϒΩϡʔ λεΫͷ࣮ߦঢ়گ ࣮ߦϩά αʔόʔΛ૿ͯ͠ ฒྻΞοϓ λεΫͷ࣮ߦڥ %PDLFSίϯςφ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλج൫ͷհ Πϯϑϥ %BUB$FOUFS %JSFDU$POOFDU 71$1FFSJOH %JHEBHNBTUFS "VSPSB %JHEBHXPSLFS "VUP4DBMJOH ɾɾɾ 4 0UIFS71$ ($4 #JH2VFSZ "-# ࣮ߦϩά -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. %JHEBHͷϫʔΫϑϩʔհ +my_task_1: sh>: "echo this task runs first." +my_task_2: sh>: "echo this task runs next." ஞ࣮࣍ߦ +run_if_param_is_true: if>: ${param} _do: echo>: ${param} == true ݅ذ +repeat: for_each>: fruit: [apple, orange] _do: echo>: ${fruit} ϧʔϓ +repeat: for_each>: fruit: [apple, orange] _parallel: true _do: echo>: ${fruit} ฒྻ࣮ߦ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. Ͳ͜Ͱ3VCZ͔ͭͬͯΔͷ͔ʁ w%JHEBHͷ3VCZΦϖϨʔλʔ w3VCZεΫϦϓτͰϫʔΫϑϩʔΛಈతʹ࡞Δ͜ͱ͕Ͱ͖Δ wIUUQTEPDTEJHEBHJPPQFSBUPSTSCIUNM -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. ෳςʔϒϧͷಉظ wFNCVMLσΟϨΫτϦҎԼͷશϑΝΠϧʹରͯ͠FNCVMLΛىಈ w3VCZͰϑΝΠϧ໊ҰཡΛऔಘ͠ɺ%JHEBHͰฒྻ࣮ߦ timezone: UTC +set_tables: require: tasks.rb rb>: set_tables +transfer_tables: for_each>: table: ${tables} _parallel: true _do: sh>: embulk run embulk/${table}.yml def set_tables tables = Dir.glob('embulk/*.yml') Digdag.env.store(tables: tables) end TBNQMFEJH UBTLTSC -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷࠩߋ৽Λႈʹ wΓ͍ͨ͜ͱ ूܭΫΤϦͷ࣮ߦ݁ՌΛಛఆͷςʔϒϧʹ࣍Ͱग़ྗͯ͠ੵΈ্͛ςʔϒϧʹ w❌ႈͰͳ͍ྫ # INSERT INTO SELECTのBigQuery版 query("集計クエリ", destination_table: 'table_name', write: 'append') w⭕ႈͳྫ table_yesterday = table('table_name_20181201') table_yesterday.copy('table_name_20181202') table_today = table('table_name_20181202') query("集計クエリ", destination_table: 'table_name_20181202', write: 'append') table_today.copy('table_name') UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF ᶃDPQZ ᶅDPQZ ᶄBQQFOE -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷࠩߋ৽Λႈʹ wҙ֎ͱΠϨΪϡϥʔύλʔϯ͕ଟ͘ɺ݅ذ͕ଟ͍ w%JHEBHͷ੍ޚߏจ͚ͩͰࣗ༝͕গͳ͍ wॲཧΛΓସ͑ΔͨΊʹ3VCZΛར༻ wHFNJOTUBMMHPPHMFDMPVEͰ($1ͷ"1*͕͑Δ UBCMF@OBNF@ UBCMF@OBNF@ UBCMF@OBNF ᶃDPQZ ᶅDPQZ ᶄBQQFOE ৽نσʔλϚʔτͷ࣌ʹ ࡢͷςʔϒϧ͕ͳ͍ 5JNF1BSUJUJPOFEςʔϒϧͷ߹ ௨ৗͷDPQZ͑ͳ͍ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔղܾ wঢ়گ͋ΔσʔλϚʔτ͕ผͷσʔλϚʔτͷूܭ݁ՌΛར༻͍ͯ͠Δ wσʔλϚʔτͷूܭॲཧʹґଘ͕ؔൃੜʂ wΓ͍ͨ͜ͱґଘؔΛຬͨͭͭ͠ɺฒྻՄೳͳݶΓ্͍͛ͨ EBUBNBSU EBUBNBSU EBUBNBSU 4&-&$5ʙ'30.EBUBNBSU +0*/EBUBNBSUʙ EBUBNBSUͷूܭΑΓઌʹूܭ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔղܾ σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔΛάϥϑ %"( ʹ͢Δ sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i) %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹׂ " # #EFQFOETPO" -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. σʔλϚʔτͷूܭͷґଘؔղܾ σʔλϚʔτͷ42-จΛղੳͯ͠ґଘؔΛάϥϑ %"( ʹ͢Δ sql.scan(/(?:FROM|JOIN)\s+`(.+?)`/i) %"(Λղੳ͍͔ͯͭ͘͠ͷฒྻ࣮ߦՄೳͳάϧʔϓʹׂ " # #EFQFOETPO" -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. STQFDͰ42-ϑΝΠϧͷνΣοΫ w42-ϑΝΠϧΛॻ͘ͷσʔλΞφϦετ wຊ൪ʹग़͔ͯ͠Βόον͕མͪΔ͜ͱ͕͠͠ wจࣈίʔυ͕4+*4 w#JH2VFSZͷXFC6*ͱ"1*Ͱͷࠩҟ wσʔλϚʔτͷґଘ͕ؔ॥ w$JSDMF$*Ͱશ42-ϑΝΠϧͷจ๏νΣοΫΛߦ͍ݕ w͋ͱɺ$JSDMF$*ͰΠϯςάϨʔγϣϯςετߦ͍ͬͯΔ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. 3VCZͬͯΈͯΑ͔ͬͨ w࠷ॳߦͷͪΐͬͱͨ͠εΫϦϓτ wͩΜͩΜංେԽ͢ΔʹͭΕͯΦϒδΣΫτࢦతͳઃܭʹ wΦϒδΣΫτࢦɾؔܕɾखଓ͖ܕͳͲͷελΠϧΛબΔ wHFNΛ͏͜ͱͰBXT($1ͷ"1*ୟ͚Δ -
Copyright © ZOZO Technologies, Inc. All Rights Reserved. Copyright ©
ZOZO Technologies, Inc. All Rights Reserved. 8FSF)JSJOH w͍ͭ·Ͱ3BJMT͍͍ͬͯͷ͔ΩϟϦΞͰΜͰ͍Δํʂʂ wͦͷ3VCZܦݧɺฐࣾͷσʔλج൫ͳΒ׆͔ͤ·͢Α IUUQTUFDI[P[PDPNSFDSVJU