Spores: Towards Function-Passing Style in the Age of Concurrency and Distribution

Functional programming (FP) is regularly touted as the way forward for bringing parallel, concurrent, and distributed programming to the mainstream. The popularity of the rationale behind this viewpoint (immutable data transformed by function application) has even lead to a number of object-oriented (OO) programming languages adopting functional features such as lambdas (functions) and thereby function closures. However, despite this established viewpoint of FP as an enabler, reliably distributing function closures over a network, or using them in concurrent environments nonetheless remains a challenge across FP and OO languages.

This talk presents an approach for more principled distributed and concurrent programming by introducing a new closure-like abstraction and type system, called spores, that can guarantee closures to be serializable, thread-safe, or even have general, custom user-defined properties. Crucially, our system is based on the principle of encoding type information corresponding to captured variables in the type of a spore. We'll see the power of the guarantees that this safe foundation for migratable closures facilitates through a case analysis touching upon several real-world frameworks such as Akka and Spark.

1. SPORES Scala Days 2014, Berlin, Germany June 17th, 2014 Heather Miller ! Towards Function-Passing Style in the Age of Concurrency & Distribution
3. AGENDA WHY WE NEED THEM WHAT THEY ARE WHAT YOU CAN DO WITH THEM ¡DEMO! SPORES {
4. AGENDA WHY WE NEED THEM WHAT THEY ARE WHAT YOU CAN DO WITH THEM ¡DEMO! SPORES { (spores in Scala & Javascript)
5. AGENDA WHY WE NEED THEM WHAT THEY ARE WHAT YOU CAN DO WITH THEM ¡DEMO! SPORES { COOLEST PART OF THE TALK SPOILER ALERT: (spores in Scala & Javascript)
6. TWO TRENDS DATA-CENTRIC APPLICATIONS FUNCTIONAL PROGRAMMING APPLY FUNCTIONS TO IMMUTABLE DATA. CLOSURE-HEAVY. CALLBACKS/REACTIVE.
7. OBSERVATION:
8. THESE ARECOMPLIMENTARY TRENDS OBSERVATION:
9. THESE ARECOMPLIMENTARY TRENDS OBSERVATION: FUNCTIONAL PROGRAMMING A BOON TO DATA-CENTRIC PROGRAMMING
10. WHY? The basic philosophy to transform immutable data by applying first-class functions. The observation that this functional style simplifies reasoning about data in parallel, concurrent, and distributed code. 1 2
11. THE OF DATA-PARALLEL POPULARITY FRAMEWORKS HENCE,
12. Spark MapReduce DISTRIBUTED Java 8’s monadic optionally-parallel collections Scala’s parallel collections Haskell’s Par Monad PARALLEL/ CONCURRENT Intel’s Concurrent Collections DATA-PARALLEL FRAMEWORKS
13. WHY SPORES ? ARE NECESSARY
14. Well, CLOSURES ARE OFTEN A SOURCE OF HEADACHES YOU CAN’T REALLY DISTRIBUTE THEM.
15. Well, CLOSURES ARE OFTEN A SOURCE OF HEADACHES NOT JUST IN SCALA OR JAVA. BUT CROSS-PARADIGM. YOU CAN’T REALLY DISTRIBUTE THEM.
16. PROBLEMS 1. Accidental capture of non-serializable
 variables (like this) 2. Language-specific compilation schemes
 that create implicit references to objects
 that are not serializable 3. transitive references that inadvertently hold
 on to excessively large object graphs
 creating memory leaks THELAUNDRY LIST W/CLOSURES
17. PROBLEMS 4. Capturing references to mutable objects,
 leading to race conditions in a concurrent
 setting. 5. Unknowingly accessing object members
 that are not constant such as methods, 
 which in a distributed setting can have 
 logically different meanings on different 
 machines. THELAUNDRY LIST W/CLOSURES CONT’D
18. SPARK motivating example: http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal: class MyCoolRddApp { val param = 3.14 val log = new Log(...) ... def work(rdd: RDD[Int]) { rdd.map(x => x + param) .reduce(...) } }
19. SPARK motivating example: http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal: class MyCoolRddApp { val param = 3.14 val log = new Log(...) ... def work(rdd: RDD[Int]) { rdd.map(x => x + param) .reduce(...) } } PROBLEM: not serializable because it captures this of type MyCoolRddApp which is itself not serializable (x => x + param)
20. AKKA/FUTURES def  receive  =  {    case  Request(data)  =>        future  {            val  result  =  transform(data)            sender  !  Response(result)        }   } Akka actor spawns future to concurrently process incoming results AKKA ACTOR SPAWNS A FUTURE TO CONCURRENTLY PROCESS INCOMING REQS NOT A STABLE VALUE! IT’S A METHOD CALL! PROBLEM: motivating example: http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
21. AKKA/FUTURES def  receive  =  {    case  Request(data)  =>        future  {            val  result  =  transform(data)            sender  !  Response(result)        }   } Akka actor spawns future to concurrently process incoming results AKKA ACTOR SPAWNS A FUTURE TO CONCURRENTLY PROCESS INCOMING REQS NOT A STABLE VALUE! IT’S A METHOD CALL! PROBLEM: motivating example: http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
22. ENTER: SPORES two types: 1 SPORES WITH TYPE CONSTRAINTS 2 MAINLINE SPORES proposed as Scala Improvement Proposal new research published at ECOOP’14
23. 6SRUHV $ 7\SH%DVHG )RXQGDWLRQ IRU &ORVXUHV LQ WKH $JH RI &RQFXUUHQF\ DQG 'LVWULEXWLRQ +HDWKHU 0LOOHU 3KLOLSS +DOOHU1 DQG 0DUWLQ 2GHUVN\ (3)/ DQG 7\SHVDIH ,QF1 ƇŠ‡ƒ–Š‡”Ŝ‹ŽŽ‡”ř ƒ”–‹Ŝ‘†‡”•›ƈɒ‡’ˆŽŜ…Š DQG ’Š‹Ž‹’’ŜŠƒŽŽ‡”ɒ–›’‡•ƒˆ‡Ŝ…‘1 $EVWUDFW )XQFWLRQDO SURJUDPPLQJ )3 LV UHJXODUO\ WRXWHG DV WKH ZD\ IRUZDUG IRU EULQJLQJ SDUDOOHO FRQFXUUHQW DQG GLVWULEXWHG SURJUDPPLQJ WR WKH PDLQVWUHDP 7KH SRSXODULW\ RI WKH UDWLRQDOH EHKLQG WKLV YLHZSRLQW KDV HYHQ OHG WR D QXPEHU RI REMHFWRULHQWHG 22 SURJUDPPLQJ ODQJXDJHV RXWVLGH WKH 6PDOOWDON WUDGLWLRQ DGRSW LQJ IXQFWLRQDO IHDWXUHV VXFK DV ODPEGDV DQG WKHUHE\ IXQFWLRQ FORVXUHV +RZHYHU GHVSLWH WKLV HVWDEOLVKHG YLHZSRLQW RI )3 DV DQ HQDEOHU UHOLDEO\ GLVWULEXWLQJ IXQF WLRQ FORVXUHV RYHU D QHWZRUN RU XVLQJ WKHP LQ FRQFXUUHQW HQYLURQPHQWV QRQHWKH OHVV UHPDLQV D FKDOOHQJH DFURVV )3 DQG 22 ODQJXDJHV 7KLV SDSHU WDNHV D VWHS WR ZDUGV PRUH SULQFLSOHG GLVWULEXWHG DQG FRQFXUUHQW SURJUDPPLQJ E\ LQWURGXFLQJ D QHZ FORVXUHOLNH DEVWUDFWLRQ DQG W\SH V\VWHP FDOOHG VSRUHV WKDW FDQ JXDUDQWHH FOR VXUHV WR EH VHULDOL]DEOH WKUHDGVDIH RU HYHQ KDYH FXVWRP XVHUGHILQHG SURSHUWLHV &UXFLDOO\ RXU V\VWHP LV EDVHG RQ WKH SULQFLSOH RI HQFRGLQJ W\SH LQIRUPDWLRQ FRU UHVSRQGLQJ WR FDSWXUHG YDULDEOHV LQ WKH W\SH RI D VSRUH :H SURYH RXU W\SH V\VWHP THIS IS ALSO RESEARCH. FOR ALL THE GORY DETAILS… see our paper accepted for publication at ECOOP’14 http://infoscience.epfl.ch/record/191239
24. 2 ENTER: SPORES two types: SPORES WITH TYPE CONSTRAINTS new research published at ECOOP’14 1 MAINLINE SPORES proposed as Scala Improvement Proposal
25. SPORES mainline WHAT ARE THEY? BEHAVIOR SMALL UNITS OF POSSIBLY MOBILE FUNCTIONAL http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
26. SPORES mainline WHAT ARE THEY? A closure-like abstraction for use in distributed or concurrent environments. GOAL: Well-behaved closures with controlled environments that can avoid various hazards. http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
27. SPORES mainline WHAT ARE THEY? A closure-like abstraction for use in distributed or concurrent environments. GOAL: Well-behaved closures with controlled environments that can avoid various hazards. POTENTIAL HAZARDS WHEN USING CLOSURES
 INCORRECTLY: • memory leaks • race conditions due to capturing mutable references • runtime serialization errors due to unintended capture of references http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
28. WHAT DO SPORES LOOK LIKE? Basic usage: val  s  =  spore  {      val  h  =  helper      (x:  Int)  =>  {          val  result  =  x  +  "  "  +  h.toString          println("The  result  is:  "  +  result)      }   } THE BODY OF A SPORE CONSISTS OF 2 PARTS 2 a closure a sequence of local value (val) declarations only (the “spore header”), and 1 http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal
29. SPORE 1. All captured variables are declared in 
 the spore header, or using capture 2. The initializers of captured variables 
 are executed once, upon creation of 
 the spore 3. References to captured variables do 
 not change during the spore’s execution vsCLOSURES ( ) A Guarantees... http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
30. SPORES&CLOSURES EVALUATION SEMANTICS: Remove the spore marker, and the code behaves as before SPORES & CLOSURES ARE RELATED: You can write a full function literal and pass it to something that expects a spore. (Of course, only if the function literal 
 satisfies the spore rules.) http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal:
31. HOW CAN YOU USE A SPORE? Ok. So. IN APIS http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal def  sendOverWire(s:  Spore[Int,  Int]):  Unit  =  ...   //  ...   sendOverWire((x:  Int)  =>  x  *  x  -­‐  2)   If you want parameters to be spores, then you can write it this way
32. Ok. So. http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal def  lookup(i:  Int):  DCollection[Int]  =  ...   val  indices:  DCollection[Int]  =  ...   ! for  {  i  <-­‐  indices              j  <-­‐  lookup(i)   }  yield  j  +  capture(i)   ! trait  DCollection[A]  {    def  map[B](sp:  Spore[A,  B]):  DCollection[B]    def  flatMap[B](sp:  Spore[A,  DCollection[B]]):  DCollection[B]   } HOW CAN YOU USE A SPORE? FOR-COMPREHENSIONS
33. Right, GET YOU? WHAT DOES ALL OF THAT
34. WHAT DOES ALL OF THAT http://docs.scala-lang.org/sips/pending/spores.html Scala Improvement Proposal GET YOU? SINCE... 
 Captured expressions are evaluated upon spore creation. ! Spores are like function values with an immutable environment. Plus, environment is specified and checked, no accidental capturing. THAT MEANS...
35. http://docs.scala-lang.org/sips/pending/spores.html Proposed for inclusion in Scala 2.11 OR, GRAPHICALLY... During execution Right after creation SPORES CLOSURES 1 2 WHAT DOES ALL OF THAT GET YOU?
36. http://docs.scala-lang.org/sips/pending/spores.html Proposed for inclusion in Scala 2.11 OR, GRAPHICALLY... During execution Right after creation SPORES CLOSURES 1 2 5 ‘a’ ? ? WHAT DOES ALL OF THAT GET YOU?
37. http://docs.scala-lang.org/sips/pending/spores.html Proposed for inclusion in Scala 2.11 OR, GRAPHICALLY... During execution Right after creation SPORES CLOSURES 1 2 5 ‘a’ 5 ‘a’ ? ? WHAT DOES ALL OF THAT GET YOU?
38. http://docs.scala-lang.org/sips/pending/spores.html Proposed for inclusion in Scala 2.11 OR, GRAPHICALLY... During execution Right after creation SPORES CLOSURES 1 2 5 ‘a’ 5 ‘a’ ? ? I’m in ur stuff draggin around ur object graf WHAT DOES ALL OF THAT GET YOU?
39. Cool.
40. Cool. SOCKET? WHAT IF I CAPTURE A
41. Cool. SOCKET? WHAT IF I CAPTURE A RESEARCH REALM OF
42. ENTER: SPORES two types: 1 MAINLINE SPORES proposed as Scala Improvement Proposal SPORES WITH TYPE CONSTRAINTS new research published at ECOOP’14 2
43. Cool. SOCKET? WHAT IF I CAPTURE A
44. Wouldn't it be nice if we could add these constraints, in a friendly, and composable way?
45. KEEP TRACK OF CAPTURED TYPES Idea: The spore macro can synthesize precise types automatically for newly created spores: Spore[Int,  ...]  {      type  Excluded  =  NoCapture[Actor]      type  Facts  =  Captured[Int]  with  Captured[ActorRef]   } w/ constraints CreatingSPORES ...at compile-time spore  {  val  x:  Int  =  list.size;  val  a:  ActorRef  =  this.sender      (y:  Int)  =>  ...   }  exclude[Actor] SYNTHESIZED TYPE: (a whitebox macro)
46. w/ constraints ComposingSPORES BASIC COMPOSITION OPERATORS (same as for regular functions) andThen compose How do we synthesize the result type of 
 s1 andThen s2? RESULT TYPE SYNTHESIZED BY andThen MACRO type member Facts takes “union” of the facts of s1 and s2 type member Excluded: conjunction of excluded types, needs to check Facts to see if possible
47. w/ constraints Example:Composing val  s1:  Spore[Int,  String]  {      type  Excluded  =  NoCapture[Actor]      type  Facts  =  Captured[Int]  with  Captured[ActorRef]   }  =  ...   val  s2:  Spore[String,  String]  {      type  Excluded  =  NoCapture[RDD[Int]]      type  Facts  =  Captured[Actor]   }   s1  andThen  s2    //  does  not  compile   SPORES
48. w/ constraints Example: SPORES Composing val  s1:  Spore[Int,  String]  {      type  Excluded  =  NoCapture[Actor]      type  Facts  =  Captured[Int]  with  Captured[ActorRef]   }  =  ...   val  s2:  Spore[String,  String]  {      type  Excluded  =  NoCapture[RDD[Int]]   }   s1  andThen  s2:  Spore[Int,  String]  {      type  Excluded  =  NoCapture[Actor]  with   NoCapture[RDD[Int]]      type  Facts  =  Captured[Int]  with  Captured[ActorRef]   }   !
49. WHAT DO TYPE CONSTRAINTS BUY US? Stronger constraints checked at compile time (not "just" basic spore rules) Frameworks can make stronger assumptions about spores created by users. Confidence in consuming, creating, and composing spores: Constraints accumulate monotonically Constraints are never lost when 
 composing spores Less brittleness.
50. SHIP WHEN WOULD I WANT TO A SPORE?
51. EXAMPLES 4 WHICH COULD BE 
 ADVANTAGEOUS HERE ARE (& lots probably more) 1 2 3 4 Move functionality to distributed (in-memory) data Hot-swapping actor behavior Shippable stream pipelines Portable closures e.g., Spark e.g., JVM to Javascript e.g., reconfigurable streams
52. Flow(text.split("\\s").toVector).            //  transform            map(line  =>  line.toUpperCase).            //  print  to  console  (can  also  use  ``foreach(println)``)            foreach(transformedLine  =>  println(transformedLine)).            onComplete(FlowMaterializer(MaterializerSettings()))  {                case  Success(_)  =>  system.shutdown()                case  Failure(e)  =>                    println("Failure:  "  +  e.getMessage)                    system.shutdown()            } SHIPPABLE STREAM 2 PIPELINES
53. SHIPPABLE STREAM 2 PIPELINES Each stage has a closure that deals with incoming streaming data However, one could imagine that it could be advantageous that each stage is on the machine that’s closest to the data Yet we still want the code of the entire pipeline to be assembled on one machine That means we have to send pipeline stages together with their closures to different machines after the pipeline has been assembled
54. HELP HOW DO SPORES ?
55. SHIPPABLE STREAM 2 PIPELINES Spores ensure that serialization doesn’t fail at runtime. Spores enable different serialization frameworks (e.g., Scala Pickling) Spores enable restricting types that are captured by each closure. e.g., if the pipeline is built by an actor, we want to ensure that the enclosing actor is never captured. ✓ ✓ ✓
56. Flow(text.split("\\s").toVector).            //  transform            map(line  =>  line.toUpperCase).            //  print  to  console  (can  also  use  ``foreach(println)``)            foreach(transformedLine  =>  println(transformedLine)).            onComplete(FlowMaterializer(MaterializerSettings()))  {                case  Success(_)  =>  system.shutdown()                case  Failure(e)  =>                    println("Failure:  "  +  e.getMessage)                    system.shutdown()            } SHIPPABLE STREAM 2 PIPELINES BEFORE
57. Flow(text.split("\\s").toVector).            //  transform            map(spore  {  line  =>  line.toUpperCase  }).            //  print  to  console  (can  also  use  ``foreach(println)``)            foreach(spore  {  transformedLine  =>  println(transformedLine)  }).            onComplete(FlowMaterializer(MaterializerSettings()))  {                case  Success(_)  =>  system.shutdown()                case  Failure(e)  =>                    println("Failure:  "  +  e.getMessage)                    system.shutdown()            } SHIPPABLE STREAM 2 PIPELINES AFTER
58. class  HotSwapActor  extends  Actor  {      import  context._          def  receive  =  {          case  HotSwap(spore)  =>              val  newBehavior:  Receive  =  {  case  msg  =>  spore(msg)  }              become(newBehavior)          case  ..      }   }   HOT-SWAPPING ACTOR 3 BEHAVIOR
59. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 Imagine you have a rich UI on a browser-based client interacting with a server. If fine-grained information has to be exchanged between client and server, then sending spores can simplify the problem. 1 Compose functions based on UI selections. 2 Send the composed function, and the server is very simple because it just applies the function. 3 No manual translation between low-level message fields to functions applied on the server. TOTALLY COMPOSABLE. EASILY EXTENDABLE.
60. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE SEARCH TOOL FOR USED CAR OFFERS. WEBSITE LETS USERS DEFINE A NUMBER OF PREFERENCES SUCH AS PRICE RANGE. WHEN ALL PREFERENCES HAVE BEEN SELECTED, SENT TO SERVER, WHICH FILTERS CARS.
61. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE SEARCH TOOL FOR USED CAR OFFERS. WEBSITE LETS USERS DEFINE A NUMBER OF PREFERENCES SUCH AS PRICE RANGE. WHEN ALL PREFERENCES HAVE BEEN SELECTED, SENT TO SERVER, WHICH FILTERS CARS. ISSUES: Message containing all user prefs complex. Extending website with new feature to filter for is complicated, code has to be changed in multiple locations: UI code encoding pref setting into message to send decoding pref setting from message received adapting server-side logic for new pref
62. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE CAN DO WITH SPORES IN A WAY WHERE ONLY UI NEEDS TO BE CHANGED! 1. Define spores that filter in code that’s shared between client & server. Each filter spore has type: Spore[(Car,  Boolean),  (Car,  Boolean)] This allows composing two filters using andThen val  filter  =  filter1  andThen  filter2 A car matches in the case where filter((car,  true))._2
63. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE Example filter spore: def  priceRange(from:  Int,  to:  Int)  =      spore  {          val  localFrom  =  from          val  localTo  =  to          (pair:  (Car,  Boolean))  =>  {              val  (car,  valid)  =  pair              (car,                  valid  &&  (car.price  >=  localFrom  &&  car.price  <  localTo))          }      }  
64. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE 2. Compose filters on client side Suppose there’s a collection “selections” which contains filter spores . Then, we simply fold it to get the composed filter: val  userPrefs  =        selections.foldLeft(idFilter)(          (f1,  f2)  =>  f1  andThen  f2      ) 3. Pickle and send the composed spore to the server
65. PORTABLE CLOSURES (E.G., BETWEEN JVM & JS) 4 EXAMPLE 3. On the server side: Unpickle and use the filter spore to churn through a potentially larget dataset. (Can even use frameworks like Spark for that!) val  userPrefs  =        received.unpickle[Spore[(Car,  Boolean),  (Car,  Boolean)]]   ! val  eligible  =  carsRdd.filter  {        car  =>  userPrefs((car,  true))._2     }   ! //  send  eligible  back  to  user  
66. WHAT’S IN THE RELEASE Alpha version hits sonatype in the next day or two. Spores implementation as described in SIP-21. Pickling integration module 
 (see github.com/scala/pickling) Support for a subset of type constraints described in the ECOOP’14 paper.
67. THANK YOU.