Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
embulkを利用した開発でちょっと困ったこと
Search
katsuyan
May 17, 2017
Technology
1
490
embulkを利用した開発でちょっと困ったこと
embulk meetup #3で発表した
embulkを利用した開発でちょっと困ったこと
katsuyan
May 17, 2017
Tweet
Share
More Decks by katsuyan
See All by katsuyan
realtime_marketing_system
katsuyan
0
2.6k
digdag-3tips
katsuyan
2
4.6k
ファッションチェックランキングRubyKaigiの裏側 / Fashion check ranking app for RubyKaigi2019
katsuyan
1
5.9k
ETLツールを1から作り直した話
katsuyan
1
320
Other Decks in Technology
See All in Technology
Understanding Go GC #coefl_go_jp
bengo4com
0
1.1k
Go で言うところのアレは TypeScript で言うとコレ / Kyoto.なんか #7
susisu
7
1.8k
JavaScript 研修
recruitengineers
PRO
3
180
Yahoo!広告ビジネス基盤におけるバックエンド開発
lycorptech_jp
PRO
1
280
認知戦の理解と、市民としての対抗策
hogehuga
0
360
RAID6 を楔形文字で組んで現代人を怖がらせましょう(実装編)
mimifuwa
1
310
広島発!スタートアップ開発の裏側
tsankyo
0
240
TypeScript入門
recruitengineers
PRO
19
6.1k
トヨタ生産方式(TPS)入門
recruitengineers
PRO
3
240
MySQL HeatWave:サービス概要のご紹介
oracle4engineer
PRO
4
1.7k
Preferred Networks (PFN) とLLM Post-Training チームの紹介 / 第4回 関東Kaggler会 スポンサーセッション
pfn
PRO
1
220
夢の印税生活 / Life on Royalties
tmtms
0
280
Featured
See All Featured
No one is an island. Learnings from fostering a developers community.
thoeni
21
3.4k
It's Worth the Effort
3n
187
28k
RailsConf 2023
tenderlove
30
1.2k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
61k
Templates, Plugins, & Blocks: Oh My! Creating the theme that thinks of everything
marktimemedia
31
2.5k
Evolution of real-time – Irina Nazarova, EuRuKo, 2024
irinanazarova
8
900
Scaling GitHub
holman
462
140k
How STYLIGHT went responsive
nonsquared
100
5.7k
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Unsuck your backbone
ammeep
671
58k
Fantastic passwords and where to find them - at NoRuKo
philnash
51
3.4k
Writing Fast Ruby
sferik
628
62k
Transcript
&NCVMLΛར༻ͨ͠։ൃ Ͱͪΐͬͱࠔͬͨ͜ͱ !LBUTVZBO
{:name “Katsuya Tajima” :twitter“@katsuyan121” :school {:name“芝浦工大” :year 4} :langs [:Clojure
:Go :Ruby] :likes [:Aikido :Shibainu]} XIPBNJ
࡞ͬͨͷ
Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZͷσʔλಉظ
Γ͍ͨ͜ͱ 3%4 .Z42- #JH2VFSZ .Z42-͔Β#JH2VFSZͷσʔλಉظ ͜ΕΛ&NCVML ͰΓ͍ͨ
FNCVMLͰͰ͖ͳ͍͜ͱ FNCVMLςʔϒϧ୯ҐͰసૹ ෳͷςʔϒϧಉظʹରԠͰ͖ͳ͍
#JH2VFSZ 3%4 .Z42- (Pϥούʔ࡞ FYFD
ຊ
ࠔͬͨ͜ͱᶃ
&NCVMLͷΤϥʔ͕TUEPVUʹ $ embulk run sample.yml 2> error.log 2017-05-15 18:16:38.400
+0900: Embulk v0.8.18 2017-05-15 18:16:50.038 +0900 [INFO] (0001:transaction): Loaded plugin embulk-input-mysql (0.8.2) 2017-05-15 18:16:52.789 +0900 [INFO] (0001:transaction): Loaded plugin embulk-output-bigquery (0.4.5) 2017-05-15 18:16:52.820 +0900 [INFO] (0001:transaction): Fetch size is 10000. Using server-side prepared statement. org.embulk.exec.PartialExecutionException: java.lang.RuntimeException: . . . $ cat error.log $ TUEPVUʹΤϥʔ
ղܾࡦ ऴྃεςʔλεΛར༻ ϩάͷৼΓ͚Λ(PͰཧ &NCVMLͷϩάͦͷ··ར༻ Τϥʔൃੜ࣌TUEFSSʹग़ྗ
ࠔͬͨ͜ͱᶄ
ಉظ࣌࣌ؒͣΕΔ FNCVMLJOQVUKECD .Z42-ϓϥάΠϯΛར༻ ϓϥάΠϯ͕ࣗಈͰ.Z42-ͷ UJNF[POFΛऔಘ͠ͳ͍͜ͱ͕ݪҼ
ಉظ࣌࣌ؒͣΕΔ in: type: mysql host: localhost user: user password:
password database: sample table: sample select: "*" default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . TBNQMFZNM Θ͔Γ͍͢Α͏ʹ "TJB5PLZP࣌ؒ ʹͦΖ͑ͯ·͢
ಉظ࣌࣌ؒͣΕΔ mysql> select * from sample; +------+----------+---------------------+ | id
| name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 19:46:08+09:00 | | 2 | sample2 | 2013-05-27 19:46:08+09:00 | | 3 | sample3 | 2013-05-27 19:46:08+09:00 | | 4 | sample4 | 2013-05-27 19:46:08+09:00 | | 5 | sample5 | 2013-05-27 19:46:08+09:00 | | 6 | sample6 | 2013-05-27 19:46:08+09:00 | | 7 | sample7 | 2013-05-27 19:46:08+09:00 | | 8 | sample8 | 2013-05-27 19:46:08+09:00 | | 9 | sample9 | 2013-05-27 19:46:08+09:00 | | 10 | sample10 | 2013-05-27 19:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΐ͏Ͳ࣌ؒ ͣΕͯΔ
Կ͔Φϓγϣϯͳ͍͔ͳ ͳΜ͔ͬͯͳ ͍ͭ͋ͬͨʂ
ղܾࡦ in: type: mysql host: localhost user: user password:
password database: sample table: sample select: "*" options: {useTimezone: true, serverTimezone: Asia/Tokyo} default_timezone: "Asia/Tokyo" column_options: create_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} update_time: {type: string, timestamp_format: "%Y-%m-%d %H:%M:%S%:z"} out: . . . ͍ͭ͜ΛՃ TBNQMFZNM
ղܾ mysql> select * from sample; +------+----------+---------------------+ | id
| name | create_time | +------+----------+---------------------+ | 1 | sample1 | 2013-05-27 10:46:08 | | 2 | sample2 | 2013-05-27 10:46:08 | | 3 | sample3 | 2013-05-27 10:46:08 | | 4 | sample4 | 2013-05-27 10:46:08 | | 5 | sample5 | 2013-05-27 10:46:08 | | 6 | sample6 | 2013-05-27 10:46:08 | | 7 | sample7 | 2013-05-27 10:46:08 | | 8 | sample8 | 2013-05-27 10:46:08 | | 9 | sample9 | 2013-05-27 10:46:08 | | 10 | sample10 | 2013-05-27 10:46:08 | +------+----------+---------------------+ $ embulk preview sample.yml +---------+-------------+---------------------------+ | id:long | name:string | create_time:timestamp | +---------+-------------+---------------------------+ | 1 | sample1 | 2013-05-27 10:46:08+09:00 | | 2 | sample2 | 2013-05-27 10:46:08+09:00 | | 3 | sample3 | 2013-05-27 10:46:08+09:00 | | 4 | sample4 | 2013-05-27 10:46:08+09:00 | | 5 | sample5 | 2013-05-27 10:46:08+09:00 | | 6 | sample6 | 2013-05-27 10:46:08+09:00 | | 7 | sample7 | 2013-05-27 10:46:08+09:00 | | 8 | sample8 | 2013-05-27 10:46:08+09:00 | | 9 | sample9 | 2013-05-27 10:46:08+09:00 | | 10 | sample10 | 2013-05-27 10:46:08+09:00 | +---------+-------------+---------------------------+ NZTRM FNCVMLQSFWJFX ͪΌΜͱಉظ͞Εͨ
+%#$ͷΦϓγϣϯ ݹ͍Φϓγϣϯ VTF5JNF[POFUSVF TFSWFS5JNF[POF"TJB5PLZP ৽͍͠Φϓγϣϯ VTF-FHBDZ%BUFUJNF$PEFGBMTF ͜Ε͚ͩͰ ղܾͰ͖ͳ͔ͬͨ
ࢀߟ FNCVMLJOQVUKECDͷ.Z42-ϓϥά ΠϯͰ͕࣌ؒ࣌ؒͣΕΔ IUUQRJJUBDPNLBUTVZBOJUFNT GEDBCBD
·ͱΊ ϩάͷग़ྗઌ֬ೝ͠·͠ΐ͏ ࣌ؒͷҋਂ͍ Φϓγϣϯશ෦ோΊ·͠ΐ͏ &NCVMLศར