HOLY SYNC: a sane approach to offline-first cross-platform data syncing

Today users expect to have their data available on every device, and to be able to view and edit it at any time without an Internet connection.
Given how common the need is, anyone who tried to approach this problem might have found out that it’s (un)surprisingly complex, and that there are very few available tools!
In this talk I want to show how we solved it at Clue, without going crazy, on both Android and iOS.

In particular:

* Who needs this, and who doesn’t.
* Couchbase Lite: an open-source, cross-platform, embedded NoSQL database.
* Transitioning from a standard SQL data model to a NoSQL one, with deep implications on how that affects every level of the stack.
* How embracing a “Git-like approach” to conflict management can be tremendously effective.
* Keeping performance smooth on Android.

Video at Droidcon Italy 2015:
https://www.youtube.com/watch?v=Yp1h3cd8dsg&lc

Keynote presentation (animated!):
https://drive.google.com/file/d/0B-g3x7VcJPIqLVlKX3doTkY4cmM/view?usp=sharing

Wallpaper "deal with git" (2880×1800):
https://drive.google.com/file/d/0B-g3x7VcJPIqbnFkRUlibUp5dlE/view?usp=sharing

Transcript

2. Eugenio Marletti @workingkills
3. 2014 fastest growing app category* *according to Google HEALTH & FITNESS
4. PERIOD TRACKERS
5. “An app that doesn’t fart rainbows.”
6. DATA SYNCING & SHARING
7. * cross-platform solution * no reinventing the wheel (possibly) * do as little as possible (not our core business) * open source and actively developed (avoid lock-in)
8. LOCAL SERVICE
9. SINGLE SERVER, ALWAYS CONNECTED
10. MULTIPLE SERVERS
11. DATA SYNCING BETWEEN DEVICES (AND PLATFORMS)
12. NOT ALWAYS CONNECTED
13. DATA SHARING BETWEEN USERS
15. in case you didn’t get the joke the first time
17. LITE embedded NoSQL database that speaks the CouchDB protocol (and is designed for mobile)
18. embedded NoSQL database that speaks the CouchDB protocol
19. * no schema > data consistency is now your responsibility * de-normalized data > until “it hurts” * views > forget queries, embrace map/reduce embedded NoSQL database
20. DOCUMENTS
21. { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
22. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
23. CONFLICTS
24. “CONFLICTS are not an error condition, they are the result of your infrastructure allowing the same dataset to be modified across disconnected systems. The introduction of such conflicts in such a topology is the expected behavior and their programmatic resolution is a core piece of application logic.”
25. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" } { "_id": "uniqueID", "_rev": "1-abc", "key": "value" }
26. { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true } { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg", "key": ":trollface:" }
27. { "_id": "uniqueID", "_rev": "2-cde", "key": "anotherValue" } { "_id": "uniqueID", "_rev": "2-efg "key": ":trollface: } { "_id": "uniqueID", "_rev": "3-ghi", "_deleted": true }
28. REPLICATION
29. PULL REPLICATION PUSH
30. * distributed system * based on versioning * conflicts are solved by appending a revision * data is pulled & pushed * wait a second…
33. QUERIES
34. NO
35. * get document by id * “query” a view * iterate through all documents
36. VIEWS
37. View phoneView = database.getView("phones"); phoneView.setMap(new Mapper() { @Override public void map(Map<String, Object> document, Emitter emitter) { List<String> phones = (List)document.get("phones"); for (String phone : phones) { emitter.emit(phone, document.get("name")); } } }, "2");
38. KEY VALUE "1" | "Genny" "123" | "Genny" "123" | "Maria" "2" | "Maria" "456" | "Genny" "666" | "Satan" "789" | "Genny"
39. * null * false, true (in that order) * numbers, in numeric order (duh) * strings, case-insensitive (all symbols sort before) * arrays (item-by-item) * maps/dictionaries (don’t use - except empty!)
40. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
41. [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ]
42. [ "2015-01-04", "blue" ] [ "2015-01-05", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-03", "red" ] [ "2015-01-03", "blue" ] [ "2015-01-01", "blue" ] [ "2015-01-02", "red" ] [ "2015-01-03" ] startKey [ "2015-01-03", { } ] endKey
43. BACKEND TROUBLE
44. DATABASE PROXY
45. TIPS AND TRICKS
46. don’t be scared of metadata
47. "$created_at" "$updated_at"
48. ids are awesome
49. type user_averages
50. type|day pill|2015-01-01
51. type|day|value tag|2015-01-01|!
52. type|value tag_list_item|stress
53. be wary of deletion
54. { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
55. { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" } { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "_deleted": true, "$updated_at": "2" }
56. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg "_deleted": true "$updated_at": "2 } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
57. { "_id": "pill|…", "_rev": "2-cde", "value": "taken", "$updated_at": "1" } { "_id": "pill|…", "_rev": "2-efg", "$removed": true, "$updated_at": "2" } { "_id": "pill|…", "_rev": "1-abc", "value": "missed", "$updated_at": "0" }
58. Android gotchas
59. * models are iOS only :( * “to trust [your data] is good, not to trust is better” * find the right time to start replications * when saving, objects get converted to json * when reading, json is parsed as Map<String,Object> * models
60. BIT.LY/HOLY-SYNC @workingkills