Towards Browser and Server Utopia with Scala.js: an example using CRDTs

Talk from Scala Days 2015, Amsterdam.

For links, see: http://underscore.io/blog/posts/2015/06/10/scalajs-scaladays.html

This session demonstrates the practical application of Scala.js using the example of a collaborative text editing algorithm, written once in Scala, but used from the JVM and JavaScript.

Scala.js is a compelling way to build JavaScript applications using Scala. It also addresses an important and related problem, namely using the same algorithm client-side and server-side.

After this session you'll have an appreciation of:

- where Scala.js can help with mixed environment projects;
- some of the gotchas you might encounter; and
- an understanding of collaborative text editing and CRDTs.

This session is relevant to anyone wanting to execute Scala code in JavaScript. In particular I'll focus on exposing Scala code for use from JavaScript, rather than a complete application written solely in Scala. This mitigates the risk of adopting Scala.JS, while still benefiting from shared code usage.

The demonstration will be based around a CRDT. CRDTs are an important class of algorithms for consistently combining data from multiple distributed clients. As such they are a great target for Scala.js: the algorithms and data-structures involved will typically need to run on browsers and servers and we'd like to avoid implementing the (moderately complex) code twice. The specific algorithm will be WOOT, a text CRDT for correctly combining changes (think: Google Docs).

Transcript

1. Scala.js 
 Towards Utopia Richard Dallaway, @d6y underscore.io
2. Make Change Easier
3. Agenda Part 1
 Scala.js introduction Part 2
 Collaborative text editing with WOOT Part 3
 SBT, calling to and from Javascript…
4. Two Ideas Gradually introduce Scala.js
 to an existing code base Distributed data types are
 fun & a great fit with Scala.js
5. 5 Minute Scala.js Intro — Part 1 —
6. scala-js.org
7. Your Scala App Here JavaScript SBT Plugin
8. Your Scala App Here Std Lib Libs, Macros… JavaScript SBT Plugin
9. Your Scala App Here Std Lib Libs, Macros… JavaScript DOM jQuery… SBT Plugin
10. Yes, all of Scala But not Java or reflection-based code
11. Yes, all of Scala But not Java or reflection-based code Ported New Typed JS Bindings Scalaz Scalatags scalajs-dom ScalaCheck uTest scalajs-jquery shapeless uPickle scalajs-react … … …
12. Five Basic Steps addSbtPlugin("org.scala-js" % "sbt-scalajs" % "0.6.3")
 
 Write Scala @JSExport what you want expose sbt> fastOptJs Use resulting .js file
13. // Scala def answer = Option(41).map(_ + 1)
14. // JavaScript (function() { var o = Option().apply(41); if (o.isEmpty()) { var answer = None() } else { var arg1 = o.get(); var answer = new Some().init(1 + arg1); }; return answer }); // Scala def answer = Option(41).map(_ + 1)
15. // JavaScript (function() { var this$1 = $m_s_Option().apply__O__s_Option(41); if (this$1.isEmpty__Z()) { var answer = $m_s_None$() } else { var arg1 = this$1.get__O(); var x$1 = $uI(arg1); var answer = new $c_s_Some().init___O(((1 + x$1) | 0)) }; return answer }); // Scala def answer = Option(41).map(_ + 1)
16. Features Fast Compiler Optimized Output (this app: 295k) Resulting JS is Fast Great interop with JS Community
17. Not a Framework
18. Collaborative App — Part 2 —
19. What we want to build
20. What we want to build
21. What we want to build http4s
22. What we want to build http4s
23. What we want to build http4s
24. What we want to build http4s What should these messages be?
25. Don’t Do This C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3
26. Don’t Do This C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T insert H @ 2
27. Don’t Do This C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T insert H @ 2 1 2 C A delete 3
28. Don’t Do This C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T insert H @ 2 1 2 C A delete 3 C H A C H T
29. Commutative
 Replicated
 Data
 Type
30. WOOT C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3
31. WOOT C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T C ≺ H ≺ A
32. WOOT C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T C ≺ H ≺ A 1 2 C A Delete T
33. WOOT C A T C A T Alice’s View Bob’s View 1 2 3 1 2 3 1 2 3 4 C H A T C ≺ H ≺ A 1 2 C A Delete T C H A C H A
34. WChar ID Prev Next Alpha Visible?
35. WChar ID Prev Next Alpha Visible? Site Clock
36. WChar ID Prev Next Alpha Visible? Site Clock ID ID
37. Algorithm a b c d a b c d a b c d a b c x d a c d a b y c d a b c d ❌
38. Algorithm a b c d a b c d a b c d a b c x d a c d a b y c d a b c d x ❌
39. Algorithm a b c d a b c d a b c d a b c x d a c d a b y c d a b c d x ／ ❌
40. Algorithm a b c d a b c d a b c d a b c x d a c d a b y c d a b c d x y ／ ❌
41. In JavaScript Algorithm ~ 200 lines, 7.5k Tests ~ 200 lines + require.js, underscore.js, jasmine
42. But now we have 
 two problems
43. We’re going to look at… Local Insert Remote Ingest
44. We’re going to look at… (WString,Char,Pos) => (WString,WChar) Local Insert Remote Ingest
45. We’re going to look at… (WString,Char,Pos) => (WString,WChar) Local Insert (WString,WChar) => WString Remote Ingest
46. Migrating to Scala.js — Part 3 —
47. What we want to build http4s
48. What we want to build http4s
49. Server WOOT Editor UI WebSocket WOOT
50. Server WOOT Editor UI WebSocket WOOT Browser JVM { {
51. Server WOOT Editor UI WebSocket WOOT Browser JVM { { JavaScript JavaScript Scala.js Scala.js Scala
52. Server WOOT Editor UI WebSocket WOOT Browser JVM { { JavaScript JavaScript Scala.js Scala.js Scala
53. Server WOOT Editor UI WebSocket Client Wrapper
54. build.sbt shared client server
55. build.sbt shared client server > compile .sjsir
 .class .class
56. build.sbt shared client server > compile .sjsir
 .class .class > fastOptJS .js
57. build.sbt shared client server > compile .sjsir
 .class .class > fastOptJS .js > fullOptJS .js
58. ScalaCheck shared client server "org.scalacheck" %%% "scalacheck" % "1.12.2" % "test"
59. ScalaCheck shared client server "org.scalacheck" %%% "scalacheck" % "1.12.2" % "test" > test [info] + Local insert preserves original text: OK, passed 250 tests. [info] + Insert order is irrelevant: OK, passed 250 tests. [info] + Inserting produces consistent text: OK, passed 250 tests. [info] + Insert is idempotent: OK, passed 250 tests. 
 ︙
60. ID Prev Next Alpha Visible? Site Clock ID ID
61. // Scala import scala.scalajs.js import js.annotation.JSExport @JSExport case class WChar( id: Id, alpha: Byte, prev: Id, next: Id, isVisible: Boolean = true) // HTML + JavaScript <script src="client-fastopt.js"> <script> var char = new WChar(...) </script>
62. // Scala import scala.scalajs.js import js.annotation.JSExport @JSExport case class WChar( id: Id, alpha: Byte, prev: Id, next: Id, isVisible: Boolean = true) // HTML + JavaScript <script src="client-fastopt.js"> <script> var char = new WChar(...) </script>
63. OK, but… case class WChar( id: Id, alpha: Byte, prev: Id, next: Id, isVisible: Boolean = true) Should be Char? Can’t this whole API be easier to use?
64. Semantics = Scala
65. There are differences
66. There are differences JavaScript has no Char toString differences …not as many as you think
67. Wrapper Solution package client import js.annotation.JSExport import woot.WString @JSExport class WootClient() { var doc = WString.empty() @JSExport def insert(s: String, pos: Int): Json = ??? @JSExport def ingest(json: Json): Unit = ??? }
68. uPickle sealed trait Operation { def wchar: WChar } case class InsertOp(override val wchar: WChar) extends Operation case class DeleteOp(override val wchar: WChar) extends Operation import upickle._ // Produce JSON val op: Operation = ??? val json = write(op) // Consume JSON Try(read[Operation](json)).map( op => ...)
69. uPickle sealed trait Operation { def wchar: WChar } case class InsertOp(override val wchar: WChar) extends Operation case class DeleteOp(override val wchar: WChar) extends Operation import upickle._ // Produce JSON val op: Operation = ??? val json = write(op) // Consume JSON Try(read[Operation](json)).map( op => ...)
70. uPickle sealed trait Operation { def wchar: WChar } case class InsertOp(override val wchar: WChar) extends Operation case class DeleteOp(override val wchar: WChar) extends Operation import upickle._ // Produce JSON val op: Operation = ??? val json = write(op) // Consume JSON Try(read[Operation](json)).map( op => ...)
71. uPickle sealed trait Operation { def wchar: WChar } case class InsertOp(override val wchar: WChar) extends Operation case class DeleteOp(override val wchar: WChar) extends Operation import upickle._ // Produce JSON val op: Operation = ??? val json = write(op) // Consume JSON Try(read[Operation](json)).map( op => ...) ["woot.InsertOp", { "wchar": { "id": ["woot.CharId", {...}], "alpha": "*", "prev": ["woot.Beginning",{}], "next": ["woot.Ending", {}]} } ]
72. @JSExport class WootClient() { var doc = WString.empty() @JSExport def insert(s: String, pos: Int): Json = ??? @JSExport def ingest(json: Json): Unit = ??? }
73. @JSExport class WootClient() { var doc = WString.empty() @JSExport def insert(s: String, pos: Int): Json = { val (op, wstring) = doc.insert(s.head, pos) doc = wstring write(op) } @JSExport def ingest(json: Json): Unit = ??? }
74. Effecting the DOM import org.scalajs.dom val element = dom.document.getElementById("editor") // DANGER! Run away. // Unsafe access to Ace’s API val ace = js.Dynamic.global.ace ace.edit("editor") .getSession() .getDocument() .setValue("We have control”) // Risk JS Uncaught type error
75. Effecting the DOM import org.scalajs.dom val element = dom.document.getElementById("editor") // DANGER! Run away. // Unsafe access to Ace’s API val ace = js.Dynamic.global.ace ace.edit("editor") .getSession() .getDocument() .setValue("We have control”) // Risk JS Uncaught type error
76. Effecting the DOM // JavaScript var updateEditor = function(s, isVisible) { var delta = convertWootToAceCoordinates(…); editor.getSession() .getDocument().applyDeltas([delta]); } jQuery(document).ready(function() { client = new client.WootClient(updateEditor); });
77. Effecting the DOM // JavaScript var updateEditor = function(s, isVisible) { var delta = convertWootToAceCoordinates(…); editor.getSession() .getDocument().applyDeltas([delta]); } jQuery(document).ready(function() { client = new client.WootClient(updateEditor); });
78. @JSExport class WootClient() { var doc = WString.empty() @JSExport def ingest(json: Json): Unit = ??? }
79. @JSExport class WootClient(f: js.Function2[String,Boolean,Unit]) { var doc = WString.empty() @JSExport def ingest(json: Json): Unit = ??? }
80. @JSExport class WootClient(f: js.Function2[String,Boolean,Unit]) { var doc = WString.empty() @JSExport def ingest(json: Json): Unit = ??? } // JavaScript var updateEditor = function(s, isVisible) {…
81. @JSExport class WootClient(f: js.Function2[String,Boolean,Unit]) { var doc = WString.empty() @JSExport def ingest(json: Json): Unit = Try(read[Operation](json)).foreach(applyOperation) def applyOperation(op: Operation): Unit = ??? }
82. @JSExport class WootClient(f: js.Function2[String,Boolean,Unit]) { var doc = WString.empty() @JSExport def ingest(json: Json): Unit = Try(read[Operation](json)).foreach(applyOperation) def applyOperation(op: Operation): Unit = { val (ops, wstring) = doc.integrate(op) // Become the updated document: doc = wstring // Side effects: ops.foreach { case InsertOp(ch) => f(ch.alpha.toString, true) case DeleteOp(ch) => f(ch.alpha.toString, false) } } }
83. def integrate(op: Operation): (Vector[Operation], WString) = op match { // - Don't insert the same ID twice: case InsertOp(c,_) if chars.exists(_.id == c.id) => (Vector.empty, this) // - Insert can go ahead if the next & prev exist: case InsertOp(c,_) if canIntegrate(op) => val (ops, doc) = integrate(c, c.prev, c.next).dequeue() (op +: ops, doc) // - We can delete any char that exists: case DeleteOp(c,_) if canIntegrate(op) => (Vector(op), hide(c)) // - Anything else goes onto the queue for another time: case _ => (Vector.empty, enqueue(op)) } @scala.annotation.tailrec private def integrate(c: WChar, before: Id, after: Id): WString = { // Looking at all the characters between the previous and next positions: subseq(before, after) match { // - when where's no option about where to insert, perform the insert case Vector() => ins(c, indexOf(after)) // - when there's a choice, locate an insert point based on `Id.<` case search: Vector[WChar] => val L: Vector[Id] = before +: trim(search).map(_.id) :+ after val i = math.max(1, math.min(L.length-1, L.takeWhile(_ < c.id).length)) integrate(c, L(i-1), L(i)) }
84. What we’ve Seen Multi-project build ✓ Wrote Scala, ran it both places ✓ Great interop (call JS, be called by JS) ✓ Dirty dirty dynamic calls ✓ Used cross-compiled libraries ✓
85. github.com/d6y/wootjs
86. github.com/d6y/wootjs [info] Compiling 2 Scala sources to ... [info] Fast optimizing woot-client-fastopt.js 2015-05-13 WootServer - Starting Http4s-blaze
87. github.com/d6y/wootjs [info] Compiling 2 Scala sources to ... [info] Fast optimizing woot-client-fastopt.js 2015-05-13 WootServer - Starting Http4s-blaze
88. Benefits? IDE support Single language Write once, run both places
89. “It’s the types, stupid”
90. Make Change Easier
91. Two Ideas Gradually introduce Scala.js
 to an existing code base Distributed data types are
 fun & a great fit with Scala.js
92. Thanks! Richard Dallaway, @d6y https://github.com/d6y/wootjs underscore.io