Upgrade to Pro — share decks privately, control downloads, hide ads and more …

On Pickles & Spores: Improving Support for Dist...

On Pickles & Spores: Improving Support for Distributed Programming in Scala

A talk about a new serialization framework for Scala, and a new idea for sending closures over the wire.

Slides from the ScalaDays 2013 talk.

Heather Miller

June 12, 2013
Tweet

More Decks by Heather Miller

Other Decks in Programming

Transcript

  1. What is it? PICKLING == SERIALIZATION == MARSHALLING very different

    from java serialization https://github.com/scala/pickling
  2. Closed! Slow! wait, why do we care? not serializable exceptions

    at runtime https://github.com/scala/pickling
  3. Closed! Slow! wait, why do we care? not serializable exceptions

    at runtime can’t retroactively make classes serializable https://github.com/scala/pickling
  4. Enter: Scala Pickling fast: Serialization code generated at compile- time

    and inlined at the use-site. https://github.com/scala/pickling
  5. Enter: Scala Pickling fast: Serialization code generated at compile- time

    and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable https://github.com/scala/pickling
  6. Enter: Scala Pickling fast: Serialization code generated at compile- time

    and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time https://github.com/scala/pickling
  7. Enter: Scala Pickling fast: Serialization code generated at compile- time

    and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! https://github.com/scala/pickling
  8. Enter: Scala Pickling fast: Serialization code generated at compile- time

    and inlined at the use-site. Flexible: Using typeclass pattern, retroactively make types serializable NO BOILERPLATE: Typeclass instances generated at compile-time pluggable formats: Effortlessly change format of serialized data: binary, JSON, invent your own! typesafe: Picklers are type-specialized. Catch errors at compile-time! https://github.com/scala/pickling
  9. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._

    https://github.com/scala/pickling scala>  import  json._ import  json._
  10. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._

    https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36)
  11. What does it look like? scala>  import  scala.pickling._ import  scala.pickling._

    https://github.com/scala/pickling scala>  import  json._ import  json._ scala>  case  class  Person(name:  String,  age:  Int) defined  class  Person scala>  Person("John  Oliver",  36) res0:  Person  =  Person(John  Oliver,36) scala>  res0.pickle res1:  scala.pickling.json.JSONPickle  =   JSONPickle({    "tpe":  "Person",    "name":  "John  Oliver",    "age":  36 })
  12. that’s just the btw, default behavior... you can really customize

    scala pickling too. https://github.com/scala/pickling
  13. Previous examples used default behavior Customizing Pickling Pickling is very

    customizable Before we can show these things,let's have a look at the building block of the framework... Generated picklers Standard pickle format Custom picklers for specific types Custom pickle format
  14. Pickler Combinators trait  Pickler[T]  {    //  returns  next  write

     position    def  pickle(arr:  Array[Byte],  i:  Int,  x:  T):  Int    //  returns  result  plus  next  read  position    def  unpickle(arr:  Array[Byte],  i:  Int):  (T,  Int) } Elegant programming pearl that comes from functional programming. A composable and "constructive" way to think about persisting data. Compose picklers for simple types to build picklers for more complicated types What is a pickler? simplified version of what’s actually used in scala- pickling https://github.com/scala/pickling
  15. https://github.com/scala/pickling Pickler Combinators We need 2 things: fully-implemented picklers for

    some basic types like primitives 1 Picklers for base types Functions that combine existing picklers to build compound picklers 2 example: combinator that takes a Pickler[T] and returns a Pickler[List[T]]
  16. Pickler Combinators Build a pickler for pairs (Int, String), combine

    an Int pickler and a String pickler val  myPairPickler  =  tuple2Pickler(intPickler,  stringPickler) What’s the type? Can we combine them automatically? Pickler[T], can pickle objects of type T def pickle(implicit pickler: Pickler[(Int, String)]) = { pickler.pickle((32, “yay!”)) } Goal: Can take intPickler and stringPickler as implicit parameters tuple2Pickler can be an implicit def https://github.com/scala/pickling
  17. Implicit Picklers case  class  Person(name:  String,  age:  Int,  salary:  Int)

    class  CustomPersonPickler(implicit  val  format:  PickleFormat)  extends  SPickler[Person]  {    def  pickle(picklee:  Person,  builder:  PBuilder):  Unit  =  {        builder.beginEntry(picklee)        builder.putField("name",  b  =>            b.hintTag(FastTypeTag.ScalaString).beginEntry(picklee.name).endEntry())        builder.putField("age",  b  =>            b.hintTag(FastTypeTag.Int).beginEntry(picklee.age).endEntry())        builder.endEntry()    } } implicit  def  genCustomPersonPickler(implicit  format:  PickleFormat)  =    new  CustomPersonPickler customize what you pickle! https://github.com/scala/pickling
  18. Pickle Format        trait  PickleFormat  {    

               type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling
  19. Pickle Format output any format!        trait  PickleFormat

     {                type  PickleType  <:  Pickle                def  createBuilder():  PBuilder                def  createReader(pickle:  PickleType,  mirror:  Mirror):  PReader            }        trait  PBuilder  extends  Hintable  {            def  beginEntry(picklee:  Any):  this.type            def  putField(name:  String,  pickler:  this.type  =>  Unit):  this.type            def  endEntry():  Unit            def  beginCollection(length:  Int):  this.type            def  putElement(pickler:  this.type  =>  Unit):  this.type            def  endCollection(length:  Int):  Unit            def  result():  Pickle        } https://github.com/scala/pickling
  20. Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format.

    toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }
  21. Pickle Format https://gist.github.com/heathermiller/5760171 example Output edn, Clojure’s data transfer format.

    talk to a clojure app toy builder implementation: scala>  import  scala.pickling._ import  scala.pickling._ scala>  import  edn._ import  edn._ scala>  case  class  Person(name:  String,  kidsAges:  Array[Int]) defined  class  Person scala>  val  joe  =  Person("Joe",  Array(3,  4,  13)) joe:  Person  =  Person(Joe,[I@3d925789) scala>  joe.pickle.value res0:  String  =  #pickling/Person  {  :name  "Joe"  :kidsAges  [3,  4,  13]  }
  22. What can be pickled? if there is an implicit of

    type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... can’t (yet): instances of inner classes, Types https://github.com/scala/pickling
  23. What can be pickled? if there is an implicit of

    type Pickler[Foo] in scope, you can pickle instances of it types for which an implicit pickler is in scope types for which our framework can generate picklers classes case classes generic classes singleton objects primitives & primitive arrays ... IMPORTANT: The implicit picklers are used in the generation! can’t (yet): instances of inner classes, Types https://github.com/scala/pickling
  24. Status Release 0.8.0 for Scala 2.10.2 No support for inner

    classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
  25. Status Scala 2.11 as target Goal: Release 0.8.0 for Scala

    2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
  26. Status Scala 2.11 as target Goal: Plan: 1.0 release within

    the next few months SIP for Scala 2.11 Integration with sbt, Spark, and Akka, ... Experiment: use Scala-pickling to speed up Scala compiler Release 0.8.0 for Scala 2.10.2 No support for inner classes, yet ScalaCheck tests Very soon: support for cyclic object graphs (for release 0.9.0) https://github.com/scala/pickling
  27. Motivation def  receive  =  {    case  Request(data)  =>  

         future  {            val  result  =  transform(data)            sender  !  Response(result)        } } Potential hazards when using closures incorrectly: • Memory leaks • Race conditions, due to capturing mutable references • Runtime serialization errors, due to unintended capture of references akka actor spawns a future to concurrently process incoming reqs not a stable value! http://docs.scala-lang.org/sips/pending/spores.html
  28. Serialization case  class  Helper(name:  String) class  Main  {    val

     helper  =  Helper("the  helper")    val  fun:  Int  =>  Unit  =  (x:  Int)  =>  {        val  result  =  x  +  "  "  +  helper.toString        println("The  result  is:  "  +  result)    } } Serialization of fun throws a NotSerializableException. Why? doesn’t extend serializable http://docs.scala-lang.org/sips/pending/spores.html
  29. Spores val s = spore { val h = helper

    (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled
  30. Spores val s = spore { val h = helper

    (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure.
  31. Spores val s = spore { val h = helper

    (x: Int) => { val result = x + " " + h.toString println("The result is: " + result) } } an alternative way to create closure-like objects, in a way where the environment is controlled The body of a spore consists of two parts: 1. a sequence of local value (val) declarations only, and 2.a closure. The closure of a spore has to satisfy the following rule. All free variables of the closure body have to be either 1. parameters of the closure, or 2.declared in the preceding sequence of local value declarations.
  32. Inspiration Spores are not limited to distributed programming Make your

    APIs safer by expecting a Spore instead of a function By requiring spores in your public APIs you can prevent users from introducing hazards, like race conditions. Make your users more happy by preventing them from shooting themselves in the foot! do you have state & asynchrony? http://docs.scala-lang.org/sips/pending/spores.html
  33. Status SIP-21 Spores: on docs.scala-lang.org/sips now! Get involved in the

    discussion! Pull request for Scala 2.11 and Akka 2.2.1 in preparation Integration with Scala-pickling planned http://docs.scala-lang.org/sips/pending/spores.html