Secsync - Utility to persit & relay end-to-end encrypted CRDTs

The why and how Secsync came to be.

It was created out of the need to have an end-to-end encrypted protocol to allow data synchronization/fetching incl. real-time updates to support local-first apps in combination with a web clients without locally stored data.

Transcript

1. Nik Graf https://mastodon.social/@nikgraf nikgraf.bsky.social @nikgraf
2. Utility to persit & relay end-to-end encrypted CRDTs
3. Demo
4. Let’s e2e encrypt CRDTs
5. Two online participants
6. What about the offline use-case?
8. How to store unenrypted Yjs / Automerge? • Store the document in a blob • Append changes and sometimes compress the doc -> on the server • We can’t do that with e2e encryption. Must happen on the client.
9. Let’s what we can do with Yjs & Automerge A B D C A B D C A B D C = =
10. Let’s encrypt each change A B D C xyz asd poi jkl encrypt
11. Best part Use Signal, MegOlm, MLS Don’t roll your own crypto!
12. Performance (250k changes) 250k changes Document Automerge 2.0.3 Yjs (update v2) Size
 (base64 encoded) 38.7MB 11.1MB Construct Document 1377ms 979ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
13. Performance (10k changes) 10k changes Document Automerge 2.0.3 Yjs (update v2) Size
 (base64 encoded) 1.44MB 0.4MB Construct Document 52ms 59ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
14. Problems • Documents can get large and take a long time to load • (decryption not even accounted for) • Messaging protocols are not a good fit • Use-case: fetch a document again and decrypt it • Solution: one encryption key?
15. Snapshots (with updates?) Automerge.save Yjs.encodeStateAsUpdate
16. Performance (Automerge 2.0.3) 10k changes Document Changes Snapshot Snapshot w/ 1000 Changes Size
 (base64 encoded) 1.44MB 0.001MB 0.15MB Construct Document 52ms 45ms 74ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
17. Performance (Automerge 2.0.3) 250k changes Document Changes Snapshot Snapshot w/ 1000 Changes Size
 (base64 encoded) 38.7MB 0.17KB 0.3MB Construct Document 1377ms 1423ms 1466ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
18. Performance (Yjs 13.5.28) 10k changes Document Changes Snapshot Snapshot w/ 1000 Changes Size
 (base64 encoded) 0.4MB 0.02MB 0.5MB Construct Document 59ms 0.8ms 247ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
19. Performance (Yjs 13.5.28) 250k changes Document Changes Snapshot Snapshot w/ 1000 Changes Size
 (base64 encoded) 11.1MB 0.21MB 0.25MB Construct Document 979ms 11ms 127ms https://github.com/SerenityNotes/naisho/tree/main/benchmarks/snapshots Numbers on a M1 Macbook 2020
20. Snapshots only (serial) xyz xyz asd jkl xyz asd • Server must ensure you have the most recent snapshot before accepting yours • Easy protocol, lots of data overhead, real-time collaboration not feasible • Requires a central service, delta fetching is easy
21. Snapshots + Updates (serial) xyz jkl asd • Server must ensure you have the most recent snapshot before accepting yours • Easy protocol, lots of overhead due retries when using real-time collaboration • Requires a central service, delta fetching is easy xyz asd poi
22. Snapshots + Updates (per client) xyz jkl asd • Server must ensure you have the most recent snapshot before accepting yours • Easy protocol, scaling issue with many clients, real-time collaboration works, UX nightmare • Works decentralized, delta fetching is tricky xyz asd poi jkl hjk
23. Snapshots (serial) + Updates (serial per client) xyz jkl asd • Server must ensure you have the most recent data before accepting your snapshot • More tricky protocol, little data overhead, real-time collaboration works well • Requires a central service, delta fetching is ok xyz asd poi jkl hjk poi
24. Desired properties • Offline collaboration - without multiple parties being online • Real-time collaboration incl. ephemeral state • Privacy • Be able to remove content - through Snapshots & Tombstones (if supported) • Snapshot History • Good loading times for fetching a document • Integrity checking!
25. Entities • Document -> ID • Snapshot -> Automerge.save, Y.encodeStateAsUpdate • Update -> Automerge change, Yjs Update • EphemeralMessage -> Yjs Awarness
26. Cryptography • Key rotation for Snapshots • Verify authors • Prevent against server leaving out updates or manipulating message • Prevent against server sending an older snapshot (reply attack) • Ignores reply attacks (snapshots, updates & ephemeral messages)
27. Cryptography Doc encryption
28. Cryptography EphemeralMessages Auth
29. Don’t roll you own crypto! (I shouldn’t do too)
30. Thanks! https://www.secsync.com/ https://github.com/serenity-kit/secsync
31. Resources • Inspiration: Snapdoc: Authenticated snapshots with history privacy in peer-to- peer collaborative editing https://martin.kleppmann.com/papers/snapdoc- pets19.pdf
32. Desired properties • Async collaboration - without multiple parties being online • Real-time collaboration incl. ephemeral state • Privacy • Be able to remove content - through Snapshots and Tombstones (if supported) • Solid Performance for loading a document • Confidentiality - content only accessible to participants • Authenticity - all changes can be verified to come from participants • Integrity - no change missing • Offline editing • Two modes: • Public Key Infrastructure based • Shared Symmetric Key (e.g. via PAKE) • Confidential and authenticated information who created which section • Meta data: not in scope for 1.0 to hide who is participating (Zero-knowledge proofs)