two-way sync

#cocoaheadskyiv

1. two-way sync Simplest implementation #cocoaheadskyiv Dima Zen @ 17.12.2015
2. Why? • Client-server application • Full offline support • “All changes have to be available online automatically” - client • Simultaneous editing from multiple clients: iOS, Android, Web What needs to be done
3. Whats available? • Backend written on Django (python) • REST API • google.com, stackoverflow.com
4. What do we need for sync?
5. sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority
6. changes tracking why?
7. remote local t to find conflicts
8. remote pull local t to find conflicts
9. remote local t { id: 12341, username: "john.doe", description: "iOS dev" } to find conflicts
10. remote local t { id: 12341, username: "john.doe", description: “hire me" } * pull to find conflicts
11. remote local t { id: 12341, username: "john.doe", description: “hire me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts
12. remote local t { id: 12341, username: "john.doe", description: “hire me" } * { id: 12341, username: "john.doe", description: "Android" } to find conflicts
13. { id: 12341, username: "john.doe", description: "iOS dev" } remote local t to find unsaved changes
14. { id: 12341, username: "john.doe", description: “hire me" } * push remote local t to find unsaved changes
15. { id: 12341, username: "john.doe", description: “hire me" } * ? push remote local t to find unsaved changes
16. changes tracking • Timestamp (lastModified: 17.12.2015T19:20:22.133145Z) • Version (revision: 245) • Status (dirty: true, deleted: false) • Combined aka Change Data Capture
17. { id: 12341, username: "john.doe", description: "iOS dev" } data structure { id: 12341, username: "john.doe", description: "iOS dev", revision: 213, modified: 12.12.2016, deleted: false, dirty: false } trackable plain
18. changelog local t changes tracking
19. changelog local t { id: 12341, revision: 213, modified: 12.12.2016, deleted: false } changes tracking
20. changelog local t pull changes tracking { id: 12341, revision: 213, modified: 12.12.2016, deleted: false dirty: true } *
21. changelog local t changes tracking { id: 12341, revision: 215, modified: 12.12.2016, deleted: false } { id: 12341, revision: 213, modified: 12.12.2016, deleted: false dirty: true } *
22. sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority
23. data authority • API is the only data authority • Client doesn't change revision nor timestamp, only via API • Timestamp management on client is hard • Client’s changelog can maintain its own revision, independent from the API’s one
24. db versioning
25. remote local without db version
26. remote pull local without db version
27. remote local without db version ? ? ? ? ? ?
28. remote local with db version
29. remote local with db version pull revision=215
30. remote local with db version {no changes}
31. db versioning • Depends on changes tracking implementation • Tied with changelog • Allows to identify changes deltas • Acts as an offset revision = 1023
32. changelog
33. changelog • Ordered log of data change records • Store records such as update or delete for specific object • Can track only fact of the particular event or full change representation (affected keys, old values, new values)
34. db version + changelog • Database.version == changelog.HEAD.version • Database.version == most recent data.version • Any data change increments database version • New change.version == Database.version + 1
35. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1
36. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=1 POST /notes/
37. POST /notes/ changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2
38. { id: 443, revision: 2, content: “First Note" } Note changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2
39. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note
40. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 25, change: update, entity: "Note", objectID: 443 } { id: 443, revision: 2, content: “First Note" } Note
41. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 }
42. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
43. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=2 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
44. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog { id: 442, revision: 1, content: “Lorem Ipsum" } data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } PUT /notes/442/
45. changelog API { id: 24, change: update, entity: "Note", objectID: 442 } db + changelog data Note revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" }
46. { id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 442, revision: 3, content: "Cocoaheads" } Note
47. { id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 25, change: update, entity: "Note", objectID: 443 } { id: 26, change: update, entity: "Note", objectID: 442 } { id: 442, revision: 3, content: "Cocoaheads" } Note
48. { id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }
49. { id: 24, change: update, entity: "Note", objectID: 442 } changelog API db + changelog data revision=3 { id: 443, revision: 2, content: “First Note" } Note { id: 442, revision: 3, content: "Cocoaheads" } Note DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 } { id: 25, change: update, entity: "Note", objectID: 443 }
50. { id: 442, revision: 3, content: "Cocoaheads" } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } DELETE /notes/442/ { id: 26, change: update, entity: "Note", objectID: 442 }
51. { id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
52. { id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
53. { id: 27, change: delete, entity: "Note", objectID: 442 } { id: 442, revision: 4, content: "Cocoaheads", deleted: true } Note { id: 25, change: update, entity: "Note", objectID: 443 } changelog API db + changelog data revision=4 { id: 443, revision: 2, content: “First Note" } Note { id: 24, change: update, entity: "Note", objectID: 442 } { id: 26, change: update, entity: "Note", objectID: 442 }
54. simple changelog, version per row, no changed data • Easy to implement • Easy to maintain • Doesn’t contain actual data deltas • No way to rollback invalid changes • Impossible to do field-by-field conflicts resolution pros cons
55. API • Client’s database version is required for any API data mutation API or deltas fetch • Data mutation from the unsynced client is rejected • Push is request-per-change via REST
56. API GET /sync-chunk/?revision=1331&limit=100 { "max_revision": 1334, "result": { "Note": { "updates": [ { "id": 812, "content": "Cocoaheads Kiev", "revision": 1332 }, { "id": 442, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 113, "revision": 1334 } ] } } }
57. API with update and delete of the same object, single page { "max_revision": 1335, "result": { "Note": { "updates": [ { "id": 443, "content": "Android Dev", "revision": 1333 } ], "deletes": [ { "id": 442, "revision": 1335 } ] } } }
58. API with update and delete of the same object, two pages { "max_revision": 1334, "result": { "Note": { "updates": [ { "id": 442, "content": "Android Dev", "revision": 1333 }, { "id": 443, "content": "CocoaHeads", "revision": 1334 } ] } } } { "max_revision": 1335, "result": { "Note": { "deletes": [ { "id": 443, "revision": 1335 } ] } } } page #0 page #1
59. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body
60. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”]
61. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true
62. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true
63. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict)
64. API PUT /notes/442/ { "id": 442, "content": "Android Dev", "revision": 1333 } Authorization: Token 14cb354ee6492f5288e49e26c99d376bcc1fa5c1 Revision: 1333 headers body if db.revision != headers[“revision”] true STATUS CODE: 409 (Conflict) false { "id": 442, "content": "Android Dev", "revision": 1334 } Revision: 1334 headers body
65. sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority
66. resolving conflicts • Client resolves any merge conflict • Merge changed data pushed to the API • Client-based resolution allows to use UI • Simplifies backend code, client becomes more complex • Graceful resolving requires additional effort • per-field changes tracking • usage of combined changes tracking on client done on client
67. resolving conflicts in dummy way pseudocode class OverwriteConflictResolvingStrategy { let object: Object let changelog: Changelog let update: ChangelogEvent? let delete: ChangelogEvent? func resolve(changes: [String: AnyObject]) { if let update = update { changelog.removeChange(update) } if let delete = delete { changelog.removeChange(delete) } } }
68. sync changes
 tracking conflicts API architecture background changelog db versioning experience data authority
69. architecture
70. Chunk Store Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog
71. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store
72. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store
73. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store
74. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 431
75. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 481
76. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 481
77. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501
78. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501
79. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501
80. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501
81. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store revision: 501
82. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store
83. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog pull Chunk Store
84. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store
85. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store
86. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store revision: 431
87. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store
88. Sync Scheduler Pull Operation Push Operation Sync Chunk Fetch Merge Chunk Merge Change Push Changelog push Chunk Store
89. merge pseudocode from Merge Operation class ChunkStore { var chunks: [SyncChunk] { return [] } } class Merge { enum EntityType { case User, Note, Notebook } let chunkStore = ChunkStore() func executeMerge() { let orderedTypes: [EntityType] = [.User, .Notebook, .Note] for type in orderedTypes { for chunk in chunkStore.chunks { mergeChunkUpdates(chunk, ofType: type) mergeChunkDeletes(chunk, ofType: type) } } } }
90. experience
91. expect unexpected termination • Design you code with restoration / preservation in mind • Preserve execution state after every sync operation (entity merged, change pushed, etc) • Delete preserved data once done if needed
92. changes push • Push changes in a serial queue • Push and Pull are mutually exclusive and can not be performed concurrently • Handle validation errors to prevent pushing queue hang (e.g. mark change as invalid in case of double validation error, etc)
93. SyncScheduler • Respect reachability and connection errors: make a delay before retry • Don’t use timer with small delay for check. It drains your battery • On Changelog update notify SyncScheduler about local state is being dirty (Observer, NotificationCenter, etc)
94. changes tracking: prefer Version over Timestamp • Integers easier to compare, store and use • Utilize microseconds to prevent data lost (yyyy.MM.dd’T’HH:mm:ss.SSSSSS’Z’) • NSDateFormatter doesn’t parse microseconds • Prefer NSDateComponents over NSDate
95. changes tracking • One place for all • Harder to implement • Requires filtering of unnecessary data. Leads to hell. • Simple implementation • Requires explicit tracking invocation • Tracks only necessary data auto manual
96. changelog as a separate database • Looks like a more robust solution • Allows you to imp. Changelog as a separate module • Doesn’t give a lot of benefit • Requires complex synchronization of Changeling’s context and Database’s context. (not available for iOS 7) • DB saving error may leave you up with inconsistent state: changelog doesn’t reflect database state.
97. deletes • Don’t delete objects immediately, use deleted flag instead • Purge deleted records on: • Delete change push • After running “full sync”
98. relational database sync • Apply updates from the top of your relationship tree • Beware of unexpected cascade deletion during deletes deltas applying • Try to omit many-to-many relationships
99. performance tuning • Move sync to the background queue / context • Use separate contexts: multiple general-purpose contexts, single sync context • Use data prefetching, batch saving
100. where to go • Add per-field sync metadata • Store changed values alongside with the change record • Client mutate objects sync data for more precise merge
101. Links • https://www.objc.io/issues/10-syncing-data/data- synchronization/ • https://dev.evernote.com/media/pdf/edam-sync.pdf • https://en.wikipedia.org/wiki/Change_data_capture • https://technet.microsoft.com/en-us/library/ bb933994(v=sql.105).aspx
102. Wants more? • https://msdn.microsoft.com/en-US/library/bb677158.aspx • http://floriankugler.com/2013/04/29/concurrent-core- data-stack-performance-shootout/ • https://en.wikipedia.org/wiki/Operational_transformation
103. questions?
105. thanks :)