Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ETLツールを1から作り直した話
Search
katsuyan
September 02, 2017
Technology
1
320
ETLツールを1から作り直した話
ETLツールを1から作り直した話(embulkを利用したETLツール)
katsuyan
September 02, 2017
Tweet
Share
More Decks by katsuyan
See All by katsuyan
realtime_marketing_system
katsuyan
0
2.6k
digdag-3tips
katsuyan
2
4.6k
ファッションチェックランキングRubyKaigiの裏側 / Fashion check ranking app for RubyKaigi2019
katsuyan
1
5.9k
embulkを利用した開発でちょっと困ったこと
katsuyan
1
490
Other Decks in Technology
See All in Technology
【OptimizationNight】数理最適化のラストワンマイルとしてのUIUX
brainpadpr
0
200
データエンジニアがクラシルでやりたいことの現在地
gappy50
3
850
AI時代の経営、Bet AI Vision #BetAIDay
layerx
PRO
1
1.7k
AIエージェントを現場で使う / 2025.08.07 著者陣に聞く!現場で活用するためのAIエージェント実践入門(Findyランチセッション)
smiyawaki0820
6
580
Unson OS|48時間で「売れるか」を判定する AI 市場検証プラットフォーム
unson
0
170
AWS re:Inforce 2025 re:Cap Update Pickup & AWS Control Tower の運用における考慮ポイント
htan
1
210
専門分化が進む分業下でもユーザーが本当に欲しかったものを追求するプロダクトマネジメント/Focus on real user needs despite deep specialization and division of labor
moriyuya
0
1k
MCP認可の現在地と自律型エージェント対応に向けた課題 / MCP Authorization Today and Challenges to Support Autonomous Agents
yokawasa
5
1.7k
Segment Anything Modelの最新動向:SAM2とその発展系
tenten0727
0
350
製造業の課題解決に向けた機械学習の活用と、製造業特化LLM開発への挑戦
knt44kw
0
150
GMOペパボのデータ基盤とデータ活用の現在地 / Current State of GMO Pepabo's Data Infrastructure and Data Utilization
zaimy
3
190
alecthomas/kong はいいぞ
fujiwara3
6
1.4k
Featured
See All Featured
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
33
2.4k
Making Projects Easy
brettharned
117
6.3k
Automating Front-end Workflow
addyosmani
1370
200k
A designer walks into a library…
pauljervisheath
207
24k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.5k
Speed Design
sergeychernyshev
32
1.1k
Navigating Team Friction
lara
188
15k
[RailsConf 2023] Rails as a piece of cake
palkan
56
5.7k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
1k
Code Reviewing Like a Champion
maltzj
524
40k
Gamification - CAS2011
davidbonilla
81
5.4k
Visualization
eitanlees
146
16k
Transcript
© 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γͨ͠ ୈճֶੜΤϯδχΞݶఆ-5େձʂʂʂ ాౡࠀ࠸
© 2017 VASILY,Inc. ࣗݾհ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”
:year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
© 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ݄ ॴࡏ౦ژ۠ޒాδχΞεϏϧ' ैۀһ໊
ࢿຊۚԯ දऔకۚࢁ༟थ औకࠓଜխઍ༿େี ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
© 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔ΒͷສΛ͑ΔΛܝࡌ ݄ؒສਓҎ্͕ར༻͢Δຊ࠷େڃͷϑΝογϣϯαΠτ
© 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ
ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɹɹϕετΠϊϕʔςΟϒΞϓϦେ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ࿈ଓϕετΞϓϦड ੈքͰ7"4*-:͚ͩ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. &5-ͱ ‣ &YUSBDU5SBOTGPSN-PBE ‣ &YUSBDUใݯ͔ΒσʔλΛநग़ ‣ 5SBOTGPSNσʔλΛมɾՃ
‣ -PBE֨ೲઌʹσʔλΛϩʔυ
© 2017 VASILY,Inc. ࠓճΓ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZͷσʔλಉظ
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ͑ΒΕ ͳ͘ͳͬͨ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞ :".- &NCVMLΛ࣮ߦͯ͠σʔλಉظ
&NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦⾪ :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLͱ ‣ σʔλసૹϑϨʔϜϫʔΫ ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ ‣ ϓϥάΠϯʹΑΓෳͷೖྗɾग़ྗʹରԠ
‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ
© 2017 VASILY,Inc. &NCVMLͷ͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM
:.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:
Ϣʔβʔ໊ password: ύεϫʔυ database: ରͷσʔλϕʔε table: ରͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfileͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
© 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦⾪ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳςʔϒϧͷಉظ ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ ‣ ॊೈͳϩάऩू
‣ Τϥʔൃੜ࣌ͷ௨ॲཧ
© 2017 VASILY,Inc. (Pϥούʔͷׂ ‣ ઃఆϑΝΠϧͷੜ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू
‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ࠷ऴߏ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML
© 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT