Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
ETLツールを1から作り直した話
Search
katsuyan
September 02, 2017
Technology
1
320
ETLツールを1から作り直した話
ETLツールを1から作り直した話(embulkを利用したETLツール)
katsuyan
September 02, 2017
Tweet
Share
More Decks by katsuyan
See All by katsuyan
realtime_marketing_system
katsuyan
0
2.6k
digdag-3tips
katsuyan
2
4.7k
ファッションチェックランキングRubyKaigiの裏側 / Fashion check ranking app for RubyKaigi2019
katsuyan
1
5.9k
embulkを利用した開発でちょっと困ったこと
katsuyan
1
500
Other Decks in Technology
See All in Technology
ソースを読むプロセスの例
sat
PRO
15
9.7k
AI時代の開発を加速する組織づくり - ブログでは書けなかったリアル
hiro8ma
1
240
「最速」で Gemini CLI を使いこなそう! 〜Cloud Shell/Cloud Run の活用〜 / The Fastest Way to Master the Gemini CLI — with Cloud Shell and Cloud Run
aoto
PRO
0
140
[VPoE Global Summit] サービスレベル目標による信頼性への投資最適化
satos
0
210
混合雲環境整合異質工作流程工具運行關鍵業務 Job 的經驗分享
yaosiang
0
140
AIフル活用で挑む!空間アプリ開発のリアル
taat
0
130
クラウドとリアルの融合により、製造業はどう変わるのか?〜クラスメソッドの製造業への取組と共に〜
hamadakoji
0
330
「魔法少女まどか☆マギカ Magia Exedra」におけるバックエンドの技術選定
gree_tech
PRO
0
100
会社を支える Pythonという言語戦略 ~なぜPythonを主要言語にしているのか?~
curekoshimizu
1
440
Dylib Hijacking on macOS: Dead or Alive?
patrickwardle
0
440
CREが作る自己解決サイクルSlackワークフローに組み込んだAIによる社内ヘルプデスク改革 #cre_meetup
bengo4com
0
270
All About Sansan – for New Global Engineers
sansan33
PRO
1
1.2k
Featured
See All Featured
GraphQLの誤解/rethinking-graphql
sonatard
73
11k
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
How To Stay Up To Date on Web Technology
chriscoyier
791
250k
Reflections from 52 weeks, 52 projects
jeffersonlam
353
21k
I Don’t Have Time: Getting Over the Fear to Launch Your Podcast
jcasabona
34
2.5k
It's Worth the Effort
3n
187
28k
A Tale of Four Properties
chriscoyier
161
23k
Designing Dashboards & Data Visualisations in Web Apps
destraynor
231
53k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
657
61k
Designing for Performance
lara
610
69k
Making Projects Easy
brettharned
120
6.4k
A designer walks into a library…
pauljervisheath
209
24k
Transcript
© 2017 VASILY,Inc. &5-πʔϧΛ͔Β࡞Γͨ͠ ୈճֶੜΤϯδχΞݶఆ-5େձʂʂʂ ాౡࠀ࠸
© 2017 VASILY,Inc. ࣗݾհ {:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大”
:year 4} :intern “VASILY,Inc.” :langs [:Clojure :Go :Ruby] :likes [:Aikido :Shibainu]}
© 2017 VASILY,Inc. ໊ࣾגࣜձࣾ7"4*-: ϰΝγϦʔ 7"4*-: *OD ઃཱ݄ ॴࡏ౦ژ۠ޒాδχΞεϏϧ' ैۀһ໊
ࢿຊۚԯ දऔకۚࢁ༟थ औకࠓଜխઍ༿େี ओཁגओ άϩʔϏεϕϯνϟʔΩϟϐλϧ ҏ౻ςΫϊϩδʔϕϯνϟʔζ (.0ϕϯνϟʔύʔτφʔζ ,%%*גࣜձࣾ גࣜձࣾߨஊࣾ
© 2017 VASILY,Inc. Ҏ্ͷϑΝογϣϯ&$αΠτ͔ΒͷສΛ͑ΔΛܝࡌ ݄ؒສਓҎ্͕ར༻͢Δຊ࠷େڃͷϑΝογϣϯαΠτ
© 2017 VASILY,Inc. Google Play ϕετΞϓϦ J04ΞϓϦ ɾ"QQ4UPSF#&45 ɾ&TTFOUJBMೝఆ "OESPJEΞϓϦ
ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɾ(PPHMF1MBZϕετΞϓϦ ɹɹϕετΠϊϕʔςΟϒΞϓϦେ ɾ(PPHMF͔Βτοϓσϕϩούʔೝఆ ࿈ଓϕετΞϓϦड ੈքͰ7"4*-:͚ͩ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. 7"4*-:ΠϯλʔϯͰ͖ͬͯͨ͜ͱ ‣ &5-πʔϧͷ࡞Γ͠ ‣ %#Ҡߦ .POHP%#ͷ໓
‣ $JSDMF$*ͷΞοϓσʔτ ‣ ΫϩʔϥʔͷҰ෦ίϯςφԽ
© 2017 VASILY,Inc. &5-ͱ ‣ &YUSBDU5SBOTGPSN-PBE ‣ &YUSBDUใݯ͔ΒσʔλΛநग़ ‣ 5SBOTGPSNσʔλΛมɾՃ
‣ -PBE֨ೲઌʹσʔλΛϩʔυ
© 2017 VASILY,Inc. ࠓճΓ͔ͨͬͨ͜ͱ 3%4 .Z42- ͔Β#JH2VFSZͷσʔλಉظ
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. چ&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ #JH2VFSZʹσʔλಉظ σʔλͷ૿Ճʹ͑ΒΕ ͳ͘ͳͬͨ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ৽&5-πʔϧ σʔλಉظͷྲྀΕ .Z42-͔ΒεΩʔϚऔಘ ઃఆϑΝΠϧΛ࡞ :".- &NCVMLΛ࣮ߦͯ͠σʔλಉظ
&NCVMLΛར༻ʂʂ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. ᶃ&NCVMLʹΑΔσʔλಉظ ͜ͷ෦⾪ :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLͱ ‣ σʔλసૹϑϨʔϜϫʔΫ ‣ 'MVFOUEͷόον൛ͷΑ͏ͳπʔϧ ‣ ϓϥάΠϯʹΑΓෳͷೖྗɾग़ྗʹରԠ
‣ ೖྗɾग़ྗ࣌ͷΤϥʔॲཧͳͲΛߦͬͯ͘ΕΔ
© 2017 VASILY,Inc. &NCVMLͷ͍ํ ‣ :".-ܗࣜͰઃఆϑΝΠϧهड़ ‣ &NCMVLΛ࣮ߦ ‣ FNCVMLSVOʙZNM
:.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVMLઃఆϑΝΠϧͷྫ in: type: mysql host: ϗετ໊ user:
Ϣʔβʔ໊ password: ύεϫʔυ database: ରͷσʔλϕʔε table: ରͷςʔϒϧ select: "*" out: type: bigquery mode: replace auth_method: json_key json_keyfile: json_keyfileͷύε project: ϓϩδΣΫτ໊ dataset: σʔληοτ໊ table: ςʔϒϧ໊ gcs_bucket: GCSͷόέοτ໊ auto_create_gcs_bucket: false compression: GZIP source_format: NEWLINE_DELIMITED_JSON .Z42-ˠ#JH2VFSZͷ߹ FNCVMLJOQVUNZTRM FNCVMLPVUQVUCJHRVFSZ
© 2017 VASILY,Inc. ᶄ(Pϥούʔ ͜ͷ෦⾪ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#:
© 2017 VASILY,Inc. &NCVML͚ͩͰ࣮ݱͰ͖ͳ͍͜ͱ ‣ ෳςʔϒϧͷಉظ ‣ ςʔϒϧʹରͯͭ͠ઃఆϑΝΠϧ͕ඞཁ ‣ ॊೈͳϩάऩू
‣ Τϥʔൃੜ࣌ͷ௨ॲཧ
© 2017 VASILY,Inc. (Pϥούʔͷׂ ‣ ઃఆϑΝΠϧͷੜ ‣ &NCVMLͷ࣮ߦ ‣ ϩάऩू
‣ Τϥʔൃੜ࣌ͷॲཧ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ࠷ऴߏ 'JMFJDPO*DPOTNBEFCZ'SFFQJLGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF :.-JDPO*DPOTNBEFCZ:BOOJDLGSPNXXXqBUJDPODPNJTMJDFOTFECZ$$#: 4BMBEJDPO*DPOTNBEFCZ.BEFCZPMJWFSGSPNXXXqBUJDPODPNJTMJDFOTFECZ'MBUJDPO#BTJD-JDFOTF
© 2017 VASILY,Inc. ͬͱศརʹ IUUQUFDIWBTJMZKQFOUSZEJHEBH@BOE@FNCVML
© 2017 VASILY,Inc. Πϯλʔϯืूத IUUQTXXXXBOUFEMZDPNQSPKFDUT