go.mercari.io/datastore はいいぞ！ / go.mercari.io/datastore is pretty good!

go.mercari.io/datastore の紹介です。
https://gaeja.connpass.com/event/90133/

Transcript

1. go.mercari.io/datastore ͸͍͍ͧʂ ajn #3 Θ͔Ί ·͞ͻΖ
2. Θ͔Ί ·͞ͻΖ @v vakame TypeScript Masahiro Wakame ٕज़ॻయ appengine/go photo from golang.org/doc/gopher/
3. GAE/SE Node.js β Ͱ΋ࠓ೔ͷ࿩͸GAE/GoͰ͢
4. ͸͡Ίʹ • ׂΓࠐΈ࣭໰׻ܴʂ • ฉ͖͍ͨ͜ͱΛฉ͍ͯ΄͍͠ • ࡞ऀͳͷͰશମ૾Λطʹ஌͍ͬͯΔ • Կ͕Θ͔ΒΜ͔Θ͔ΒΜͷͩ… ࢲ͕࡞ऀͰ͢
5. go.mercari.io/datastore ࠓ೔ͷ࿩୊Ͱ͢
6. ઌ೔v1.0.0ϦϦʔε ҆ఆͯ͠Δͥʂ
7. ͦΕԿʁ
8. ͦΕԿʁ • (AppEngine|Cloud) Datastore Wrapper • In Go! Seamless! • Middleware • Retry, Caching, Logging… etc • Batch operation • Single-Op Batch-Op aggregation
9. ઃܭࢥ૝ • Cloud DatastoreͷAPIʹͳΔ΂͘دͤΔ • ༨ܭͳ͜ͱ͸ͳΔ΂͘͠ͳ͍ • ErrFieldMismatchແࢹͷσϑΥϧτԽ • RunInTransactionͷࣗಈRetryͷ࡟আ • PropertyLoadServerʹctx͋͛ͨ • Batchૢ࡞͸଍ͨ͠
10. (AppEngine|Cloud) Datastore
11. ͦ΋ͦ΋ Datastoreͬͯ2छྨ͋Δɻ ͦͷ͜ͱΛ஌͍ͬͯ·͔͢ʁ
12. Datastore Service AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore GAE Instance Any Server RPC Endpoint Package Runtime Env
13. Datastore Storage AppEngine RPC Cloud RPC google.golang.org/ appengine/datastore cloud.google.com/ go/datastore GAE Instance Any Server RPC Endpoint Package Runtime Env ଘࡏ͢Δػೳʹ ݁ߏ͕ࠩ͋Δ ৽ چ
14. ͡Ό͋৽࢖͑͹Α͘Ͷʁ • GAE/SE Go͔Βͷར༻ʹ੍໿͋Γ • Socket API͕ඞཁ… ΊΜͲ͍… • Cloudͷํɺͨ·ʹτϥϒͬͯΔ • GCPUG Slack #g-datastore_ja ࢖͍෼͚͕·ͩ·ͩඞཁͩʂ
15. ޓ׵ੑ… • (AppEngine|Cloud) Datastore͸… • ཪͷ࣮૷͸ಉ͡ • ஌ݟ΍ઃܭ͸9ׂҎ্࢖͍ճͤΔ • ͨͩ͠ɺAPI͕ҟͳΔʂ • ͳΜͳΒpackage΋׬શʹผ
16. ΑΖ͍͠ʂ ͳΒ͹Wrapperͩʂ
17. Wrapperͷಛ௃ • Cloud Datastore(৽)ͷAPI͕ϕʔε • ࣮૷ͷ(AppEngine|Cloud)ࠩସՄ • goonϢʔβͷͨΊʹboom΋͋Γ·͢
18. package༡ཡ • godoc root • godoc aedatastore • godoc clouddatastore
19. ͋ΓಘΔະདྷ • AppEngine→GKE΁ͷࡌͤସ͑ • ཧ࿦্ΠέΔؾ͕͢Δ… • ࣄྫ͸ࣾ಺Ͱ΋ແ͍໛༷ • gVisorϕʔεͷGAE/SE૿͑ͯΔͬΆ͍ • GAE/Go΋কདྷϫϯνϟϯඍཻࢠ…ʁ
20. ҠߦLIVE CODING • ΛɺޙͰ΍Δ͕࣌ؒ͋Δͱ͍͍ͳ͊
21. • datastore:”,flatten” λάͷଘࡏ • Cloud DatastoreͷAPIʹ͋ΘͤͯΔ • ίʔυͷॻ͖׵͚͑ͩͰ͸μϝ • structͷλά෇͚Λݟ௚͢ʂ ςετ͸͔ͬ͠Γʂ ಛʹ஫ҙ͢΂͖ࠩ෼
22. flatten? • datastore.Entity ʹ͍ͭͯ • AE → Entity͸Entity࣋ͯͳ͍ • Cloud → Entity͸Entity࣋ͯΔ • struct͸Entityʹม׵͞ΕΔ • nest ͨ͠struct͸EntityͷೖΕࢠʹͳΔ
23. flatten? type AEEntity struct { "Slice.A" []string "Slice.B" []string } ͜ͷίʔυྫ͸ ΠϝʔδͰ͢ type Inner struct { A string B string } type Data struct { Slice []Inner } type CloudEntity struct { Slice []struct{ A string B string } }
24. flatten? type Inner struct { A string B string } type Data struct { Slice []Inner } type Data struct { Slice []Inner ↩ `datastore:",flatten"` } flattenͰAEͱಉ͡ʹ
25. ☠ ATTENTION ☠ • Cloud DatastoreͰNestͨ͠Entity࡞Δͱ AppEngine Datastore͔ΒಡΊͳ͍ • ͸ͣ(ࢼͯ͠ͳ͍) • ؾܰͳؾ࣋ͪͰAppEngine͔ΒGetͯ͠ CloudʹPut͢Δͱࢮ͵Մೳੑ͕͋Δ ςετ͸͔ͬ͠ΓͶʂ
26. • KeyLoader etc ͷCloudଆಠ઎ػೳ։์ • UnitTestͰCloud Datastore Emulator • GoLandͰσόοΨ࢖͍΍͍͢ • Local͔Βσʔλ౤ೖ͕༰қ ͔͠͠΍ͬͨ͜ͱ͸ͳ͍ʂ ศརͳ఺
27. Middleware
28. ͳͥཉ͍͔͠ • Middlewareͷ༻్ • RPCෆௐ࣌ͷϦτϥΠ • MemcacheͳͲͷར༻ • ϩάग़ྗ • େྔEntity Put࣌ͷRPC਺෼ׂ
29. ΞϓϦͷ࣮૷ʹ ूதͯ͐͠ʂ
30. Middlewareʹ ΍ΒͤΑ͏ʂ
31. ࠓ͋ΔMiddleware • AE Memcache • In-Memory Cache • Automatic RPC Retry • Redis(Memorystore) Cache • Logger, Chaos RPC, Fishbone … • ↑ओʹςετ΍ݕূ༻ ຊ൪ར༻࣮੷ͷน
32. Batch Operation
33. ͜͏͍͏ίʔυ͕ॻ͚Δ type Comment struct { Message string } type Post struct { Content string CommentIDs []int64 `json:"-"` Comments []*Comment `datastore:"-"` } // Entityͷ४උ(ׂѪ) ……… // start fetching... posts := make([]*Post, 0) _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! bt := client.Batch() for _, post := range posts { comments := make([]*Comment, 0) for _, id := range post.CommentIDs { comment := &Comment{} bt.Get(client.IDKey("Comment", id, nil), comment, nil) comments = append(comments, comment) } post.Comments = comments } err = bt.Exec(ctx) if err != nil { panic(err) }
34. ΋͠΋Batch͕ͳ͔ͬͨΒ // start fetching... posts := make([]*Post, 0) _, err = client.GetAll(ctx, client.NewQuery("Post").Order("Content"), &posts) if err != nil { panic(err) } // Let's batch get! var commentKeys []datastore.Key for _, post := range posts { for _, commentID := range post.CommentIDs { commentKeys = append(commentKeys, client.IDKey("Comment", commentID, nil)) } } comments := make([]*Comment, len(commentKeys)) err = client.GetMulti(ctx, commentKeys, comments) if err != nil { panic(err) } for _, post := range posts { for _, commentID := range post.CommentIDs { for idx, comment := range comments { key := commentKeys[idx] if commentID == key.ID() { post.Comments = append(post.Comments, comment) break } } } } ErrNoSuchEntity͕ ࠞͬͨ͟Β͞ΒʹͭΒ͍…ʂ
35. ErrNoSuchEntityରԠ bt.Get(key, comment, func(err error) error { if err == datastore.ErrNoSuchEntity { // ignore ErrNoSuchEntity return nil } else if err != nil { return err } post.Comments = append(post.Comments, comment) return nil }) Entity୯ҐͷΤϥʔϋϯυϥ༗Γ
36. ҆ఆ͍ͯ͠Δ͔
37. ར༻࣮੷༗ • ϝϧΧϦ Ξος (AppEngine) • ϝϧΧϦ KYC (Cloud) • ٕज़ॻయ (AppEngine) • timakin͞Μ (AppEngine) ར༻ใࠂ଴ͬͯ·͢❤
38. όάݕग़ 1݅ͷΈʂ༏लʂ
39. ςετεΠʔτ • Goͷαϒςετͷ࢓૊Έ͕࠷ߴ • https://blog.golang.org/subtests • େྔͷςετΛ༻ҙ͢Δ • go.mercari.io/datastore/testsuite • AE, Cloud Ͱಉ͡ςετΛͿΜճ͢ • ↑ޓ׵ੑ
40. Ҡߦखॱ ཁ͢Δʹطଘͱͷࠩ෼
41. جຊతͳํ਑ • API͸Cloudͱ΄΅ಉ౳ • ϝιου໊ͱ͔ • iterator.Done Λ࢖͏ͱ͔ • datastore.MultiError ͱ͔ • struct͸interfaceʹஔ͖׵͑ • AEͱCloudͰ࣮૷͕ҟͳΔͨΊ
42. godocݟͯ͘Εʂ How to migrate to this library Here's an overview of what you need to do to migrate your existing code. replace *datastore.Key to datastore.Key. replace *datastore.Query to datastore.Query. replace *datastore.Iterator to datastore.Iterator. from AE Datastore import go.mercari.io/datastore and go.mercari.io/datastore/aedatastore both. rewrite those using functions of datastore package to FromContext function and Client method calls. replace err.(appengine.MultiError) to err.(datastore.MultiError) . Stop using appengine.BlobKey and replace with string. replace google.golang.org/appengine/datastore.Done to google.golang.org/api/iterator.Done . replace key.IntID() to key.ID() . replace key.StringID() to key.Name() . When nesting a struct, apply `datastore:", flatten "` to the corresponding field. Delete datastore.TransactionOptions, it is not supported. If using google.golang.org/appengine/datastore , replace to go.mercari.io/datastore . from Cloud Datastore import go.mercari.io/datastore and go.mercari.io/datastore/clouddatastore . rewrite those using functions of datastore package to FromContext function and Client method calls. replace *datastore.Commit to datastore.Commit . If using cloud.google.com/go/datastore , replace to go.mercari.io/datastore . from goon to boom replace *goon.Goon to *boom.Boom . replace goon.FromContext(ctx) to ds, _ := aedatastore.FromContext(ctx); boom.FromClient(ctx, ds) . ଍Γͳ͍ͷݟ͚ͭͨΒIssue΁ʂ
43. ҠߦLIVE CODING github.com/vvakame/ucon-todo
44. Welcome to contribution!
45. ͚ͨͯ͢❤ • ɹɹɹɹɹɹ໨ࢦͤGitHub300←❤ • GCPUG SlackͰ࣭໰͢Δˡ❤❤ • Θ͔Βͳ͍͜ͱΛIssueʹ͢Δˡ❤❤❤ • ࢖ͬͯϒϩάͱ͔ॻ͘ˡ❤❤❤❤ • Pull RequestΛ͘ΕΔˡ❤❤❤❤❤❤❤ https://gcpug.jp/join
47. ۙگ ๻ͷपΓͰSpanner࢖͏ͱ͜૿͖͑ͯͯͯ΍͹͍ go.mercari.io/datastore ͷग़൪͕…(´ɾТɾ`) ͱ͍͏Θ͚ͰSpannerͷΫΤϦύʔαॻ͖࢝Ί·ͨ͠