Upgrade to Pro — share decks privately, control downloads, hide ads and more …

changelogと戦う

Avatar for oprst oprst
March 29, 2018

 changelogと戦う

#eureka_meetup

Avatar for oprst

oprst

March 29, 2018
Tweet

More Decks by oprst

Other Decks in Programming

Transcript

  1. CONFIDENTIAL - Toreta Inc., All Right Reserved. ࣗ ݾ ঺

    հ w ໊લɿࢁޱকԝ .BTBDIJLB:BNBHVDIJ  w UXJUUFSɿ!UZBLBDIJLB w ৬छɿσʔλؔ࿈ԿͰ΋԰ w ৬ྺɿ w *OOPWB4USVDUVSF d  w %"56.456%*0 d  w 5PSFUB ʙ ॳͷࣄۀձࣾ ຊ΋ॻ͍ͯ·͢ ࣗݾ঺հ
  2. CONFIDENTIAL - Toreta Inc., All Right Reserved. A g e

    n d a  ࣄۀ঺հ  ݱঢ়ͷσʔλΠϯϑϥ  DIBOHFMPHͱͷઓ͍
  3. CONFIDENTIAL - Toreta Inc., All Right Reserved. I n t

    r o d u c t i o n ൟ੝ళΦʔφʔͨͪͷҙݟΛ΋ͱʹɺ ຊ౰ʹҿ৯ళͷͨΊʹͳΔπʔϧΛ࡞Δ ͱ͍͏໨తͰઃཱͨ͠ձࣾͰ͢
  4. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .

    ࣄ ۀ ঺ հ ҿ৯ళ ͓٬༷ τϨλ wࡾํΑ͠ wϑΣΞͳαʔϏε wҿ৯ళͷݱ৔ͷຯํ ࡾํΑ͠
  5. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .

    ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ "QQଆ ෼ੳ༻؀ڥ MySQL #JH2VFSZ mysql_to_ bigquery 3FQMJDBUJPO "DDFTTMPHܥ &YQPSU GCS #BUDI 'JMUFS  &YUSBDU %BUBMBC %BUBqPX -PBE "OBMZ[F σʔλΠϯϑϥ
  6. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .

    ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ ؀ڥߏங%FQMPZ EJHEBH4FSWFSͷߏ੒ MPDBM
  7. CONFIDENTIAL - Toreta Inc., All Right Reserved. 0 2 .

    ݱ ঢ় ͷ σ ʔ λ Π ϯ ϑ ϥ Schedule.dig 501 1BSFOU $IJME import_to_bigquery/main.dig bigquery_to_salesforce/main.dig execute_query/main.dig backup/main.dig ga/main.dig incremental/main.dig sync_event/main.dig DBMM …… ※Ұ෦ൈਮ DBMM EJHEBH4FSWFSʹ ొ࿥͢Δͷ͸͚ͩ͜͜ EJHEBHͰͷλεΫͷ؅ཧ
  8. 0 2 . ݱ ঢ় ͷ σ ʔ λ Π

    ϯ ϑ ϥ MPDBM UBTLͷ௥Ճ͸ɺ13ϕʔεͰ 13 13 13
  9. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ https://www.slideshare.net/ssuser9c738a/bigquery-84528574 bqsushi #6ʹͯ ଍Γͳ͍σʔλͰChangelogͷ࿩Λ͍ͯͨ͠
  10. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ 3FTFSWBUJPOͷDSFBUFVQEBUFΠϕϯτΛ τϦΨʔʹ4JEFLJRͷKPCΛ࣮ߦ kinesis FWFOUσʔλ KTPO ΛQVU $MPVE8BUDI MBNCEB S3 4DIFEVMF࣮ߦ 3FDPSETͷऔಘ TDIFEVMF࣮ߦ औಘͨ͠3FDPSETΛ ϑΝΠϧʹ·ͱΊͯQVU Ұ෦ͷςʔϒϧ͸ɺऔ͍ͬͯͨ gcsͷεέδϡʔϧసૹ
  11. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͜Εεέʔϧͤ͞Δͷେมʜ
  12. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͱΓ͍ͨσʔλ͕૿͑Δͨͼʹɺ αʔόʔαΠυͷ޻਺Λऔͬͯ͠·͏ɻ
  13. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ σʔλଆͷཁ๬Ͱ ຊ൪Λॏͨ͘͘͠ͳ͍
  14. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ Ͳ͏͔ͨ͠
  15. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ .Z42-ͷCJOMPH࢖͓͏ ͜͜Λ׆༻͠Α͏
  16. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ CJOMPHͱ͸ʁ
  17. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ όΠφϦϩάʹ͸ɺςʔϒϧ࡞੒ૢ࡞΍ςʔϒϧσʔλ΁ͷมߋͳͲͷσʔ λϕʔεมߋΛهड़͢ΔʮΠϕϯτʯ͕֨ೲ͞Ε·͢ɻ·ͨɺߦϕʔεͷϩ Ϊϯά͕࢖༻͞ΕΔ৔߹Λআ͖ɺ(Ұக͢Δߦͷͳ͍ DELETE ͳͲͷ) જࡏ తʹมߋΛߦ͓͏ͱͨ͠εςʔτϝϯτʹ͍ͭͯͷΠϕϯτ΋֨ೲ͞Ε·͢ Ҿ༻: https://dev.mysql.com/doc/refman/5.6/ja/binary-log.html binlog
  18. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͜Ε͸࢖͑ͦ͏
  19. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ௐࠪ
  20. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ EPDLFSʹNBTUFSTMBWFͷ NZTRM؀ڥΛ࡞ͬͯDPOGJHपΓͷௐࠪ
  21. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ Θ͔ͬͨ͜ͱ
  22. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ [mysqld] log-bin=/var/log/mysql/bin-log binlog_format = ROW server-id=2 log-slave-updates ※ৄࡉ͸ׂѪ͠·͢ɻ ROWʹ͠ͳ͍ͱߦຖͷมߋ͕औΕͳ͍ slaveͰ΋binlogΛు͖ग़͢ TMBWFͷNZDPOG
  23. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ CJOMPHͷαϯϓϧ ͜ΕΛ#JH2VFSZʹऔΓࠐΊΔܗʹ͍ͨ͠
  24. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ IUUQTHJUIVCDPN[BMPSB CJOMPHQBSTFS ྑ͛͞ͳ3FQPΛ'PSLͯ͠վम ※golang੡ CJOMPHΛKTPOʹͯ͠ు͖ग़͠
  25. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͋ͱ͸औΓࠐΉ͚ͩʂ
  26. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͱ͸͍͔ͳ͍
  27. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ݸਓ৘ใ
  28. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ CJOMPHʹ͸ɺݸਓ৘ใ͕ͷͬͨ·· #JH2VFSZʹͦͷ··-PBEͰ͖ͳ͍ɻ
  29. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ QBSTFͨ͠CJOMPH͸ɺ ೔໿(#
  30. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ࠓޙαʔϏε͕੒௕͢Δ൐ͬͯɺ σʔλྔ͕૿͑Δ
  31. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ ͳΔ΂͘Πϯϑϥͷ؅ཧΛͤͣʹɺ େྔͷσʔλΛߴ଎ʹॲཧ͍ͨ͠
  32. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ $MPVE%BUBGMPX
  33. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ͸ σʔλύΠϓϥΠϯΛͭͳ͛ͯɺ σʔλΛม׵֦ͯ͠ॆ͢Δ ϑϧϚωʔδυαʔϏε
  34. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ $MPVE%BUBGMPXͱ͸ "QBDIF#FBNͰ ॻ͍ͨσʔλύΠϓϥΠϯͷ ࣮ߦ؀ڥ SVOOFS Λ ఏڙͯ͘͠ΕΔ
  35. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ Cloud Dataflowͱ͸ runnerΛ`DirectRunner `ʹ͢Ε͹ɺ localͰ࣮ߦͰ͖Δɻ
  36. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯͷ։ൃ %BUBMBC
  37. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ "QBDIF#FBNͷύΠϓϥΠϯGPS%BUBGMPX %BUBMBC %FW DirectRunner 5FTU DataflowRunner 5FTU $IBOHF "EE5BTL
  38. 0 3 . c h a n g e l

    o g ͱ ઓ ͏ %BUBGMPXͰ΍͍ͬͯΔ͜ͱ ɾݸਓ৘ใͷ࡟আɺNBTL ɾి࿩൪߸ɺࢯ໊ɺϝϞཝͳͲ ɾ#JH2VFSZͰऔΓճ͠ܗʹม׵ SPXʹมߋ͕͋ͬͨߦ͢΂͕ͯೖͬͯ͘Δ
  39. CONFIDENTIAL - Toreta Inc., All Right Reserved. wαʔόʔαΠυͷ޻਺Λ࢖͏͜ͱͳ͘ɺ DIBOHFMPH͕औΕΔΑ͏ʹͳͬͨ w%BUBqPXศརɻॲཧ଎౓͕஗͔ͬͨΒউ

    खʹBVUP4DBMJOHͯ͘͠ΕΔ w"QBDIF#FBNͷσόοΫ͕݁ߏ໘౗ष͍ w#JH2VFSZʹऔΓࠐΜ͚ͩͲɺࠓͷϑΥʔ Ϛοτͩͱ࢖͍ͮΒ͍ʜ ·ͱΊ