On Pickles & Spores: Improving Support for Distributed Programming in Scala

A talk about a new serialization framework for Scala, and a new idea for sending closures over the wire.

Slides from the ScalaDays 2013 talk.

1. pICKLES &sPORES HEATHER MILLER On IMPROVING SCALA’S SUPPORT FOR DISTRIBUTED PROGRAMMING
2. plug: http://lampwww.epfl.ch/~hmiller/scala2013/
3. pICKLES &sPORES HEATHER MILLER On IMPROVING SCALA’S SUPPORT FOR DISTRIBUTED PROGRAMMING
4. with: PHILIPP HALLER EUGENE BURMAKO MARTIN ODERSKY TYPESAFE EPFL EPFL/TYPESAFE
5. wHAT IS THIS TALK ABOUT?
6. wHAT IS THIS TALK ABOUT? Making distributed programming easier in scala
7. This kind of distributed system
8. This kind of distributed system insert social network of your choice here ALso!
9. Bottomline: Machines Communicating
10. Bottomline: Machines Communicating How can we simplify distribution at the language-level?
11. sCALA pICKLING Agenda macros & typeclasses Spores
12. sCALA pICKLING Agenda Spores
13. Pickles! scala.pickling https://github.com/scala/pickling
14. What is it? https://github.com/scala/pickling
15. What is it? PICKLING == SERIALIZATION == MARSHALLING https://github.com/scala/pickling
16. What is it? PICKLING == SERIALIZATION == MARSHALLING very different from java serialization https://github.com/scala/pickling
17. wait, why do we care? https://github.com/scala/pickling
18. Slow! wait, why do we care? https://github.com/scala/pickling
19. Closed! Slow! wait, why do we care? https://github.com/scala/pickling
20. Closed! Slow! wait, why do we care? not serializable exceptions at runtime https://github.com/scala/pickling
21. Closed! Slow! wait, why do we care? not serializable exceptions at runtime can’t retroactively make classes serializable https://github.com/scala/pickling
22. Enter: Scala Pickling https://github.com/scala/pickling
23. Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. https://github.com/scala/pickling
24. Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable https://github.com/scala/pickling
25. Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time https://github.com/scala/pickling
26. Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! https://github.com/scala/pickling
27. Enter: Scala Pickling fast: Serialization code generated at compile- time and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! typesafe: Picklers are type-specialized. Catch errors at compile-time! https://github.com/scala/pickling
28. What does it look like? https://github.com/scala/pickling
29. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling
30. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._
31. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36)
32. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._ https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36) scala>  res0.pickle res1:  scala.pickling.json.JSONPickle  =   JSONPickle({    "tpe":  "Person",    "name":  "John  Oliver",    "age":  36 })
33. and... it’s pretty fast https://github.com/scala/pickling
34. collections: Time Benchmarks
35. Benchmarks collections: free Memory (more is better)
36. Benchmarks collections: size
37. Benchmarks geotrellis: time
38. Benchmarks evactor: time Java runs out of memory
39. Benchmarks evactor: time (no java, more events)
40. that’s just the btw, default behavior... https://github.com/scala/pickling
41. that’s just the btw, default behavior... you can really customize scala pickling too. https://github.com/scala/pickling
42. Previous examples used default behavior Customizing Pickling Pickling is very customizable Before we can show these things,let's have a look at the building block of the framework... Generated picklers Standard pickle format Custom picklers for specific types Custom pickle format
43. Pickler Combinators trait  Pickler[T]  {    //  returns  next  write  position    def  pickle(arr:  Array[Byte],  i:  Int,  x:  T):  Int    //  returns  result  plus  next  read  position    def  unpickle(arr:  Array[Byte],  i:  Int):  (T,  Int) } Elegant programming pearl that comes from functional programming. A composable and "constructive" way to think about persisting data. Compose picklers for simple types to build picklers for more complicated types What is a pickler? simplified version of what’s actually used in scala- pickling https://github.com/scala/pickling
44. https://github.com/scala/pickling Pickler Combinators We need 2 things: fully-implemented picklers for some basic types like primitives 1 Picklers for base types Functions that combine existing picklers to build compound picklers 2 example: combinator that takes a Pickler[T] and returns a Pickler[List[T]]
45. Pickler Combinators Build a pickler for pairs (Int, String), combine an Int pickler and a String pickler val  myPairPickler  =  tuple2Pickler(intPickler,  stringPickler) What’s the type? Can we combine them automatically? Pickler[T], can pickle objects of type T def pickle(implicit pickler: Pickler[(Int, String)]) = { pickler.pickle((32, “yay!”)) } Goal: Can take intPickler and stringPickler as implicit parameters tuple2Pickler can be an implicit def https://github.com/scala/pickling
46. Implicit Picklers case  class  Person(name:  String,  age:  Int,  salary:  Int) class  CustomPersonPickler(implicit  val  format:  PickleFormat)  extends  SPickler[Person]  {    def  pickle(picklee:  Person,  builder:  PBuilder):  Unit  =  {        builder.beginEntry(picklee)        builder.putField("name",  b  =>            b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())        builder.putField("age",  b  =>            b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())        builder.endEntry()    } } implicit  def  genCustomPersonPickler(implicit  format:  PickleFormat)  =    new  CustomPersonPickler customize what you pickle! https://github.com/scala/pickling
47. Pickle Format        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling
48. Pickle Format output any format!        trait  PickleFormat  {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling
49. Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format. toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }
50. Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format. talk to a clojure app toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }
51. What can be pickled? if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... can’t (yet): instances of inner classes, Types https://github.com/scala/pickling
52. What can be pickled? if there is an implicit of type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... IMPORTANT: The implicit picklers are used in the generation! can’t (yet): instances of inner classes, Types https://github.com/scala/pickling
53. Status Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
54. Status Scala 2.11 as target Goal: Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
55. Status Scala 2.11 as target Goal: Plan: 1.0 release within the next few months SIP for Scala 2.11 Integration with sbt, Spark, and Akka, ... Experiment: use Scala-pickling to speed up Scala compiler Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
56. And now onto something completely different.
57. spores! scala http://docs.scala-lang.org/sips/pending/spores.html
58. What are they?
59. small units of mobile functional behavior What are they?
60. http://docs.scala-lang.org/sips/pending/spores.html proposed for inclusion in scala 2.11 small units of mobile functional behavior What are they?
61. First, some Motivation http://docs.scala-lang.org/sips/pending/spores.html
62. Motivation def  receive  =  {    case  Request(data)  =>        future  {            val  result  =  transform(data)            sender  !  Response(result)        } } Potential hazards when using closures incorrectly: • Memory leaks • Race conditions, due to capturing mutable references • Runtime serialization errors, due to unintended capture of references akka actor spawns a future to concurrently process incoming reqs not a stable value! http://docs.scala-lang.org/sips/pending/spores.html
63. Serialization case  class  Helper(name:  String) class  Main  {    val  helper  =  Helper("the  helper")    val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {        val  result  =  x  +  "  "  +  helper.toString        println("The  result  is:  "  +  result)    } } Serialization of fun throws a NotSerializableException. Why? doesn’t extend serializable http://docs.scala-lang.org/sips/pending/spores.html
64. Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled
65. Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure.
66. Spores val s = spore { val h = helper (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure. The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either 1. parameters of the closure, or 2.declared in the preceding sequence of local value declarations.
67. Inspiration Spores are not limited to distributed programming Make your APIs safer by expecting a Spore instead of a function By requiring spores in your public APIs you can prevent users from introducing hazards, like race conditions. Make your users more happy by preventing them from shooting themselves in the foot! do you have state & asynchrony? http://docs.scala-lang.org/sips/pending/spores.html
68. Status SIP-21 Spores: on docs.scala-lang.org/sips now! Get involved in the discussion! Pull request for Scala 2.11 and Akka 2.2.1 in preparation Integration with Scala-pickling planned http://docs.scala-lang.org/sips/pending/spores.html
69. qUESTIONS ? heather.miller@epfl.ch @heathercmiller Contact