Upgrade to Pro — share decks privately, control downloads, hide ads and more …

DatastoreからSpannerに 移行したいぞ途中編 / migrate Datasto...

DatastoreからSpannerに 移行したいぞ途中編 / migrate Datastore to Spanner in progress

GCPUG Tokyo Spanner Day August 2021
https://gcpug-tokyo.connpass.com/event/220499/

DatastoreからSpannerに 移行したいぞ途中編 という話

Masahiro Wakame

August 06, 2021
Tweet

More Decks by Masahiro Wakame

Other Decks in Programming

Transcript

  1. ΞʔΩςΫνϟ2 1. API (GraphQL ) 2. Service (domain ) 3.

    Repository (DB) LayeredͬΆ͍͚ͲશવͦΜͳ͜ͱͳ͍
  2. ΍ͬͯΈͨ Traceࠐͷ࣮ߦ݁Ռ 2021-08-05T10:42:25.1409165Z === RUN TestSpanne r 2021-08-05T10:42:25.2019180Z 2021/08/05 10:42:25

    Duration: 68.2807ms Name: google.spanner.admin.database.v1.DatabaseAdmin.CreateDatabase Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2030092Z ParentSpanID: 66d0f65276a786dc SpanID: 41a185b1034c5e57 ChildSpanCount: 0 2021-08-05T10:42:25.2034069Z INFO: create spanner client with projects/tbf-test/instances/tbf-test-instance/databases/for-ut-a76 3 2021-08-05T10:42:25.2035431Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_utils.go:7 0 2021-08-05T10:42:25.2037531Z 2021/08/05 10:42:25 Duration: 179.701µs Name: cloud.google.com/go/spanner.NewClient Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2038616Z ParentSpanID: 66d0f65276a786dc SpanID: 1c72141091f035d2 ChildSpanCount: 0 2021-08-05T10:42:25.2050051Z 2021/08/05 10:42:25 Duration: 2.842616ms Name: google.spanner.v1.Spanner.BatchCreateSessions Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2051837Z ParentSpanID: d21332cdab39e5c7 SpanID: ade4c02b39debc42 ChildSpanCount: 0 2021-08-05T10:42:25.2054728Z 2021/08/05 10:42:25 Duration: 2.925417ms Name: cloud.google.com/go/spanner.BatchCreateSessions Code: 0 Message: Attributes: map[] Annotations: [***Time:2021-08-05 10:42:25.201796419 +0000 UTC m=+157.694579279 Message:Creating a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204703536 +0000 UTC m=+157.697486496 Message:Received a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204718536 +0000 UTC m=+157.697501396 Message:Finished creating 2 sessions Attributes:map[]*** ] 2021-08-05T10:42:25.2056889Z ParentSpanID: 0000000000000000 SpanID: d21332cdab39e5c7 ChildSpanCount: 1 2021-08-05T10:42:25.2058866Z 2021/08/05 10:42:25 Duration: 410.303µs Name: google.spanner.v1.Spanner.BeginTransaction Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2060645Z ParentSpanID: f742a36e1e950d4d SpanID: 88b54f8ac68294bd ChildSpanCount: 0 2021-08-05T10:42:25.2061510Z INFO: before ReadRo w 2021-08-05T10:42:25.2062577Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2 0 2021-08-05T10:42:25.2064464Z 2021/08/05 10:42:25 Duration: 9.8µs Name: cloud.google.com/go/spanner.Read Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2065690Z ParentSpanID: f742a36e1e950d4d SpanID: 6386dee853276c38 ChildSpanCount: 1 2021-08-05T10:42:25.2067399Z 2021/08/05 10:42:25 Duration: 555.803µs Name: google.spanner.v1.Spanner.StreamingRead Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2069446Z ParentSpanID: 3e576d47e1cb43b3 SpanID: 1928fca56e701b2e ChildSpanCount: 0 2021-08-05T10:42:25.2071376Z 2021/08/05 10:42:25 Duration: 748.605µs Name: cloud.google.com/go/spanner.RowIterator Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2072630Z ParentSpanID: 6386dee853276c38 SpanID: 3e576d47e1cb43b3 ChildSpanCount: 1 2021-08-05T10:42:25.2074888Z WARNING: err: spanner: code = "NotFound", desc = "row not found(Table: Organization, PrimaryKey: (\"111\")) " 2021-08-05T10:42:25.2076611Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2 9 2021-08-05T10:42:25.2078426Z INFO: before Apply with ApplyAtLeastOnc e 2021-08-05T10:42:25.2081087Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:4 9 2021-08-05T10:42:25.2084536Z 2021/08/05 10:42:25 Duration: 533.803µs Name: google.spanner.v1.Spanner.Commit Code: 10 Message: Transaction 2 aborted due to active transaction 1. The emulator only supports one transaction at a time. Some best practices to avoid ABORT errors in Cloud Spanner service are : 2021-08-05T10:42:25.2087265Z 1. Avoid use of nested transactions . 2021-08-05T10:42:25.2088255Z 2. Explicitly Rollback failed transactions . 2021-08-05T10:42:25.2090414Z 3. All transactions should be running inside of retry loops . 2021-08-05T10:42:25.2091644Z Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2092587Z ParentSpanID: f4f88a04fc14f3a8 SpanID: cfc9196389b9ca23 ChildSpanCount: 0
  3. ΍ͬͯΈͨ ͦͯ͠Timeout΁… 2021-08-05T10:42:25.1409165Z === RUN TestSpanne r 2021-08-05T10:42:25.2019180Z 2021/08/05 10:42:25

    Duration: 68.2807ms Name: google.spanner.admin.database.v1.DatabaseAdmin.CreateDatabase Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2030092Z ParentSpanID: 66d0f65276a786dc SpanID: 41a185b1034c5e57 ChildSpanCount: 0 2021-08-05T10:42:25.2034069Z INFO: create spanner client with projects/tbf-test/instances/tbf-test-instance/databases/for-ut-a76 3 2021-08-05T10:42:25.2035431Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_utils.go:7 0 2021-08-05T10:42:25.2037531Z 2021/08/05 10:42:25 Duration: 179.701µs Name: cloud.google.com/go/spanner.NewClient Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2038616Z ParentSpanID: 66d0f65276a786dc SpanID: 1c72141091f035d2 ChildSpanCount: 0 2021-08-05T10:42:25.2050051Z 2021/08/05 10:42:25 Duration: 2.842616ms Name: google.spanner.v1.Spanner.BatchCreateSessions Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2051837Z ParentSpanID: d21332cdab39e5c7 SpanID: ade4c02b39debc42 ChildSpanCount: 0 2021-08-05T10:42:25.2054728Z 2021/08/05 10:42:25 Duration: 2.925417ms Name: cloud.google.com/go/spanner.BatchCreateSessions Code: 0 Message: Attributes: map[] Annotations: [***Time:2021-08-05 10:42:25.201796419 +0000 UTC m=+157.694579279 Message:Creating a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204703536 +0000 UTC m=+157.697486496 Message:Received a batch of 2 sessions Attributes:map[]*** ***Time:2021-08-05 10:42:25.204718536 +0000 UTC m=+157.697501396 Message:Finished creating 2 sessions Attributes:map[]*** ] 2021-08-05T10:42:25.2056889Z ParentSpanID: 0000000000000000 SpanID: d21332cdab39e5c7 ChildSpanCount: 1 2021-08-05T10:42:25.2058866Z 2021/08/05 10:42:25 Duration: 410.303µs Name: google.spanner.v1.Spanner.BeginTransaction Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2060645Z ParentSpanID: f742a36e1e950d4d SpanID: 88b54f8ac68294bd ChildSpanCount: 0 2021-08-05T10:42:25.2061510Z INFO: before ReadRo w 2021-08-05T10:42:25.2062577Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2 0 2021-08-05T10:42:25.2064464Z 2021/08/05 10:42:25 Duration: 9.8µs Name: cloud.google.com/go/spanner.Read Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2065690Z ParentSpanID: f742a36e1e950d4d SpanID: 6386dee853276c38 ChildSpanCount: 1 2021-08-05T10:42:25.2067399Z 2021/08/05 10:42:25 Duration: 555.803µs Name: google.spanner.v1.Spanner.StreamingRead Code: 0 Message: Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2069446Z ParentSpanID: 3e576d47e1cb43b3 SpanID: 1928fca56e701b2e ChildSpanCount: 0 2021-08-05T10:42:25.2071376Z 2021/08/05 10:42:25 Duration: 748.605µs Name: cloud.google.com/go/spanner.RowIterator Code: 0 Message: Attributes: map[] Annotations: [ ] 2021-08-05T10:42:25.2072630Z ParentSpanID: 6386dee853276c38 SpanID: 3e576d47e1cb43b3 ChildSpanCount: 1 2021-08-05T10:42:25.2074888Z WARNING: err: spanner: code = "NotFound", desc = "row not found(Table: Organization, PrimaryKey: (\"111\")) " 2021-08-05T10:42:25.2076611Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:2 9 2021-08-05T10:42:25.2078426Z INFO: before Apply with ApplyAtLeastOnc e 2021-08-05T10:42:25.2081087Z /home/runner/work/TechBookFestWeb/TechBookFestWeb/server/domains/spanner_test.go:4 9 2021-08-05T10:42:25.2084536Z 2021/08/05 10:42:25 Duration: 533.803µs Name: google.spanner.v1.Spanner.Commit Code: 10 Message: Transaction 2 aborted due to active transaction 1. The emulator only supports one transaction at a time. Some best practices to avoid ABORT errors in Cloud Spanner service are : 2021-08-05T10:42:25.2087265Z 1. Avoid use of nested transactions . 2021-08-05T10:42:25.2088255Z 2. Explicitly Rollback failed transactions . 2021-08-05T10:42:25.2090414Z 3. All transactions should be running inside of retry loops . 2021-08-05T10:42:25.2091644Z Attributes: map[Client:true FailFast:true] Annotations: [ ] 2021-08-05T10:42:25.2092587Z ParentSpanID: f4f88a04fc14f3a8 SpanID: cfc9196389b9ca23 ChildSpanCount: 0
  4. TextTraceExporter ͜Μ͚ͩ type textTraceExporter struct{ } var _ trace.Exporter =

    (*textTraceExporter)(nil ) func NewTextTraceExporter() trace.Exporter { return &textTraceExporter{ } } func (exporter *textTraceExporter) ExportSpan(sd *trace.SpanData) { if os.Getenv("DUMP_TEST_TRACE") != "TRUE" { retur n } if sd == nil { retur n } rawlog.Printf ( "Duration: %s Name: %s Code: %d Message: %s Attributes: %+v Annotations: %+v\n\tParentSpanID: %s SpanID: %s ChildSpanCount: %d" , sd.EndTime.Sub(sd.StartTime), sd.Name, sd.Code, sd.Message, sd.Attributes, sd.Annotations , sd.ParentSpanID, sd.SpanID, sd.ChildSpanCount , ) }
  5. ͓·͚ Spannerͷઃఆྫ spanner.ClientConfig { // ઀ଓͷઃఆΛߦ͏ // DUMP_TEST_TRACE Λ TRUE

    ʹͨ͠ͱ͖ɺBatchCreateSessions, BeginTransaction, DeleteSession ͱ͔ͷtrace͕ΊͪΌΊͪΌग़ΔͷͰ ਺Λগͳ͍ͨ͘͠ SessionPoolConfig: spanner.SessionPoolConfig { MinOpened: 2, // RWTx + ApplyAtLeastOnc e MaxOpened: 5 , MaxIdle: 0 , MaxBurst: 5 , WriteSessions: 0.5 , HealthCheckWorkers: 2 , HealthCheckInterval: 0 , TrackSessionHandles: true , } , },
  6. ࣗ࡞πʔϧ • sq b • Datastore likeͳॻ͖ຯͷDMLϏϧμ • si g

    • Goͷstruct͔ΒSQL૊Έཱͯ༻ͷίʔυΛੜ੒͢Δ • sid x • DDL͔ΒIndex৘ใΛ࣋ͭίʔυΛੜ੒͢Δ • cursor(Ծ ) • Datastore likeͳcursorੜ੒ϥΠϒϥϦ ݱঢ়ະެ։