Instant Pickles: Generating Object-Oriented Pickler Combinators for Fast and Extensible Serialization

Presented at OOPSLA'13, Indianapolis, IN, USA
Full publication: http://infoscience.epfl.ch/record/187787/files/oopsla-pickling_1.pdf

Serialization or pickling, i.e., persisting runtime objects by converting them into a binary or text representation is ubiquitous in distributed programming. Pickler combinators are a popular approach from functional programming designed to alleviate some of the tedium of writing pickling code by hand, but they don't translate well to object-oriented programming due to qualities like open class hierarchies and subtyping polymorphism. Furthermore, both functional pickler combinators and Java-based serialization frameworks tend to be tied to a specific pickle format, leaving programmers no choice of how their data is persisted. In this paper, we present object-oriented pickler combinators and a framework for generating them at compile-time, designed to be the default serialization mechanism of the Scala programming language. Our framework is extensible; (1) using Scala's implicit parameters, users can add their own easily-swappable pickle format, (2) using the type class pattern, users can provide their own custom picklers to override the default behavior of the Scala pickling framework. In addition to extensibility and need for little to no boilerplate, the static generation of our OO picklers achieves a factor 6 speedup over Java Serialization, and performs on par or up to 3 times faster than popular "fast" Java serialization frameworks like Kryo.

Transcript

1. Pickles Heather Miller heather.miller@epfl.ch : instant Generating Object-Oriented Pickler Combinators for Fast and Extensible Serialization Philipp Haller Eugene Burmako Martin Odersky OOPSLA’13, Indianapolis, IN, USA October 30th, 2013
2. M v
3. M v More and more traditional applications are moving to the cloud or to mobile devices. many applications are becoming distributed
4. M v More and more traditional applications are moving to the cloud or to mobile devices. many applications are becoming distributed These applications run on a variety of distributed systems
5. this kind of distributed system a cluster of commodity machines churning through a massive data-parallel job
6. insert social network of your choice here This kind of distributed system ! a smartphone interacting with a service running in the cloud
7. communication is central
8. communication is central Serialization is central to communication
9. communication is central interoperability between different services increasing in importance. Serialization is central to communication
10. communication is central interoperability between different services increasing in importance. many message formats required often within the same application Serialization is central to communication
11. communication is central Efficient communication required. interoperability between different services increasing in importance. many message formats required often within the same application Serialization is central to communication
12. solved problem? B ,isn’t serialization a
13. https://github.com/scala/pickling Java and others all have built-in support for serialization.
14. https://github.com/scala/pickling Java and others all have built-in support for serialization. why do we care?
15. https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?
16. https://github.com/scala/pickling S w! The overhead of Java serialization nearly always dominates other communication costs. “ ” Object Serialization for Marshalling Data in a Java Interface to MPI Carpenter et al. Serialization slows communication by two orders of magnitude “ ” Java and others all have built-in support for serialization. why do we care?
17. https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care? Java Serialization is still slow. Industry knows this. jvm- serialization- group
18. https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?
19. C ! https://github.com/scala/pickling S w! Java and others all have built-in support for serialization. why do we care?
20. /p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling Enter:
21. /p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling FAST Enter:
22. /p 1 new approach to serialization Compile-time pickler generation. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
23. /p 1 new approach to serialization Compile-time pickler generation. json binary xml protobuf or, invent your own format! https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
24. /p 1 new approach to serialization Compile-time pickler generation. Customize how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
25. new scala module 3 (Incubating.) Increasing adoption, community libraries, external contributors. /p 1 new approach to serialization Compile-time pickler generation. Customize how objects are pickled. https://github.com/scala/pickling FAST open, extensible 2 Serialize to any format. Enter:
28.                       
29.                        ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL
30.                        ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Some Topics Covered in the Paper Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing
31.                        ,QVWDQW 3LFNOHV *HQHUDWLQJ 2EMHFW2ULHQWHG 3LFNOHU &RPELQDWRUV IRU )DVW DQG ([WHQVLEOH 6HULDOL]DWLRQ +HDWKHU 0LOOHU (3)/ 6ZLW]HUODQG ?2i?2`XKBHH2`!2T~X+? 3KLOLSS +DOOHU 7\SHVDIH ,QF T?BHBTTX?HH2`!ivT2b72X+QK (XJHQH %XUPDNR (3)/ 6ZLW]HUODQG 2m;2M2X#m`KFQ!2T~X+? 0DUWLQ 2GHUVN\ (3)/ 6ZLW]HUODQG K`iBMXQ/2`bFv!2T~X+? $EVWUDFW $V PRUH DSSOLFDWLRQV PLJUDWH WR WKH FORXG DQG DV ³ELJ GDWD´ HGJHV LQWR HYHQ PRUH SURGXFWLRQ HQYLURQPHQWV WKH SHUIRU PDQFH DQG VLPSOLFLW\ RI H[FKDQJLQJ GDWD EHWZHHQ FRPSXWH QRGHVGHYLFHV LV LQFUHDVLQJ LQ LPSRUWDQFH $Q LVVXH FHQWUDO WR GLVWULEXWHG SURJUDPPLQJ \HW RIWHQ XQGHUFRQVLGHUHG LV VHULDOL]DWLRQ RU SLFNOLQJ LH SHUVLVWLQJ UXQWLPH REMHFWV E\ FRQYHUWLQJ WKHP LQWR D ELQDU\ RU WH[W UHSUHVHQWDWLRQ 3LFN OHU FRPELQDWRUV DUH D SRSXODU DSSURDFK IURP IXQFWLRQDO SUR JUDPPLQJ WKHLU FRPSRVDELOLW\ DOOHYLDWHV VRPH RI WKH WHGLXP RI ZULWLQJ SLFNOLQJ FRGH E\ KDQG EXW WKH\ GRQ¶W WUDQVODWH ZHOO WR REMHFWRULHQWHG SURJUDPPLQJ GXH WR TXDOLWLHV OLNH RSHQ FODVV KLHUDUFKLHV DQG VXEW\SLQJ SRO\PRUSKLVP )XUWKHUPRUH ERWK IXQFWLRQDO SLFNOHU FRPELQDWRUV DQG SRSXODU -DYDEDVHG VHULDOL]DWLRQ IUDPHZRUNV WHQG WR EH WLHG WR D VSHFLILF SLFNOH IRUPDW OHDYLQJ SURJUDPPHUV ZLWK QR FKRLFH RI KRZ WKHLU GDWD LV SHUVLVWHG ,Q WKLV SDSHU ZH SUHVHQW REMHFWRULHQWHG SLFNOHU FRPELQDWRUV DQG D IUDPHZRUN IRU JHQHUDWLQJ WKHP DW FRPSLOHWLPH FDOOHG VFDODSLFNOLQJ GHVLJQHG WR EH WKH GHIDXOW VHULDOL]DWLRQ PHFKDQLVP RI WKH 6FDOD SURJUDPPLQJ ODQJXDJH 7KH VWDWLF JHQHUDWLRQ RI 22 SLFNOHUV HQDEOHV VLJ QLILFDQW SHUIRUPDQFH LPSURYHPHQWV RXWSHUIRUPLQJ -DYD DQG .U\R LQ PRVW RI RXU EHQFKPDUNV ,Q DGGLWLRQ WR KLJK SHUIRU PDQFH DQG WKH QHHG IRU OLWWOH WR QR ERLOHUSODWH RXU IUDPH ZRUN LV H[WHQVLEOH XVLQJ WKH W\SH FODVV SDWWHUQ XVHUV FDQ SURYLGH ERWK  FXVWRP HDVLO\ LQWHUFKDQJHDEOH SLFNOH IRU &DWHJRULHV DQG 6XEMHFW 'HVFULSWRUV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &ODVVLILFDWLRQV ± PXOWLSDUDGLJP ODQ JXDJHV REMHFWRULHQWHG ODQJXDJHV DSSOLFDWLYH IXQFWLRQDO ODQJXDJHV ' >3URJUDPPLQJ /DQJXDJHV@ /DQJXDJH &RQVWUXFWV DQG )HDWXUHV ± LQSXWRXWSXW .H\ZRUGV 6HULDOL]DWLRQ SLFNOLQJ PHWDSURJUDPPLQJ GLV WULEXWHG SURJUDPPLQJ 6FDOD  ,QWURGXFWLRQ :LWK WKH JURZLQJ WUHQG WRZDUGV FORXG FRPSXWLQJ DQG PR ELOH DSSOLFDWLRQV GLVWULEXWHG SURJUDPPLQJ KDV HQWHUHG WKH PDLQVWUHDP $V PRUH DQG PRUH WUDGLWLRQDO DSSOLFDWLRQV PL JUDWH WR WKH FORXG WKH GHPDQG IRU LQWHURSHUDELOLW\ EHWZHHQ GLIIHUHQW VHUYLFHV LV DW DQ DOOWLPH KLJK DQG LV LQFUHDVLQJ $W WKH FHQWHU RI LW DOO LV FRPPXQLFDWLRQ :KHWKHU ZH FRQVLGHU D FOXVWHU RI FRPPRGLW\ PDFKLQHV FKXUQLQJ WKURXJK D PDVVLYH GDWDSDUDOOHO MRE RU D VPDUWSKRQH LQWHUDFWLQJ ZLWK D VRFLDO QHWZRUN DOO DUH ³GLVWULEXWHG´ MREV DQG DOO VKDUH WKH QHHG WR FRPPXQLFDWH LQ YDULRXV ZD\V LQ PDQ\ IRUPDWV HYHQ ZLWKLQ WKH VDPH DSSOLFDWLRQ $ FHQWUDO DVSHFW WR WKLV FRPPXQLFDWLRQ WKDW KDV UHFHLYHG VXUSULVLQJO\ OLWWOH DWWHQWLRQ LQ WKH OLWHUDWXUH LV WKH QHHG WR VHUL DOL]H RU SLFNOH REMHFWV LH WR SHUVLVW LQPHPRU\ GDWD E\ FRQ YHUWLQJ WKHP WR D ELQDU\ WH[W RU VRPH RWKHU UHSUHVHQWDWLRQ $V PRUH DQG PRUH DSSOLFDWLRQV HYROYH WKH QHHG WR FRPPXQL Topics Covered in this presentation Overview/Usage Object-Oriented Picklers Generation & Composition of OO Picklers Experimental Evaluation Picklers in Scala Pickling & Subtyping Operational Semantics Model of Inheritance Generation Algorithm Pickle Formats Runtime Picklers Object Identity & Sharing
32. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ Pickler Basics Extensibility
33. https://github.com/scala/pickling W ? pickling:
34. https://github.com/scala/pickling W ? import scala.pickling._ pickling:
35. https://github.com/scala/pickling W ? import scala.pickling._ import json._ pickling:
36. https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) pickling:
37. https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) scala> p.pickle res1: scala.pickling.json.JSONPickle = JSONPickle({ "tpe": "Person", "name": "John Oliver", "age": 36 }) pickling:
38. https://github.com/scala/pickling W ? import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:
39. https://github.com/scala/pickling W ? val pckl = p.pickle import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:
40. https://github.com/scala/pickling W ? val pckl = p.pickle scala>  pckl.unpickle[Person] res0:  Person  =  Person(John  Oliver,36) import scala.pickling._ import json._ case class Person(name: String, age: Int) val p = Person("John Oliver", 36) unpickling:
41. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ Pickler Basics Extensibility
42. collections: Time B r size [#elements] scala.collection.immutable.Vector ... scala-pickling Kryo 2 pickler-comb unsafe-pickler-comb
43. B r collections: free Memory (more is better) size [#elements] scala.collection.immutable.Vector ...
44. B r collections: size size [#elements] scala.collection.immutable.Vector ... scala-pickling Kryo
45. B r evactor: time Java runs out of memory [#events] String Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String
46. B r evactor: time (no java, more events) [#events] String Int String case  class  DataEvent  (            val  id:  String,      val  timestamp:  Int,      val  message:  String)   String Int String String Int String String Int String String Int String String Int String
47. 6000 8000 10000 12000 14000 0 5 10 15 20 25 30 35 40 Number of Wikipedia Nodes Time [ms] Wikipedia Cyclic Object Graph, Pickle Only 6 0 5 10 15 20 25 30 35 40 Time [ms] W Java Kryo v2 Scala Pickling (a) B r wikipedia: time (pickle only) cyclic object graph final  class  Vertex(    val  label:  String,      var  neighbors:  List[Vertex])
48. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics Extensibility
49. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } basics
50. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() basics
51. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() inserted by the framework basics
52. https://github.com/scala/pickling P r Picklers are first-class objects. A pickler for some type T is an instance of a concrete subclass of the following trait: trait Pickler[T] { def pickle(obj: T, builder: PickleBuilder): Unit } pickle is a macro, therefore p.pickle is expanded as follows: val  pickler:  Pickler[Person]  =  ??? val  builder  =  pickleFormat.createBuilder() pickler.pickle(p,  builder) builder.result() imported pickle format basics
53. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility
54. https://github.com/scala/pickling W b subtyping? Example: class Position(val title: String, val person: Person) What if we want to pickle Position, and Person has subclasses? dynamically, the person field could point to an instance of a subclass of Person of course... A modularly re-usable pickler for type Person must be able to pickle all possible subtypes of Person
55. https://github.com/scala/pickling S b : Not all picklers are required to handle all possible subtypes. Dynamic PIcklers exist to handle subtypes. trait  DynamicPickler[T]  {    def  pickle(obj:  T,  builder:  PickleBuilder):  Unit } Contract: pickles any object of type T (including subtypes of T) the solution
56. https://github.com/scala/pickling S b :how it works Within pickle, select Pickler[S] based on the dynamic type of the picklee: class  Person class  Firefighter  extends  Person   picklee  match  {    case  OfClass[Person]  =>  personPickler    case  OfClass[Firefighter]  =>  ffPickler    ...    case  p:  Pickleable  =>  p.pickler    case  _  =>  RuntimePickler.for(picklee.getClass)   }
57. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔
58. GeneratingP r case class Person(name: String, age: Int) Let’s generate a pickler for Job case class Job(p: Person, title: String) Scala/pickling recursively generates and composes type-class instances that represent picklers at compile-time. essence Rather than diving into type- classes, let’s just develop some intuition...
59. GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Let’s generate a pickler for Job
60. GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
61. GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
62. GeneratingP r case class Person(name: String, age: Int) case class Job(p: Person, title: String) graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] stringPickler intPickler type sought pickler Let’s generate a pickler for Job
63. GeneratingP r graphical Intuition Pickler[Job] Pickler[String] Pickler[Int] Pickler[Person] Pickler[String] Pickler[String] intPickler type sought pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
64. GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] Pickler[String] intPickler type sought pickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
65. GeneratingP r graphical Intuition Pickler[Job] Pickler[Person] type sought pickler stringPickler stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
66. GeneratingP r graphical Intuition Pickler[Job] type sought pickler stringPickler personPickler stringPickler intPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String) Let’s generate a pickler for Job
67. GeneratingP r graphical Intuition Pickler[Job] type sought pickler personPickler stringPickler intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)
68. GeneratingP r graphical Intuition type sought pickler jobPickler personPickler stringPickler intPickler stringPickler case class Person(name: String, age: Int) case class Job(p: Person, title: String)
69. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔ ✔
70. 1. Picklers are implicit values that are looked up by type (at compile time) 2. Custom picklers: implicit picklers with higher precedence than generated picklers customP r jobPickler myPersonPickler stringPickler intPickler stringPickler generated pickler custom user pickler
71. customP F r class  JSONPickleFormat  extends  PickleFormat  {  ...  } package  object  json  {    implicit  val  pickleFormat  =  new  JSONPickleFormat } same principle as custom picklers
72. Motivation Usage Performance Pickler Generation/Composition Subtyping Summary/Contributions O ✔ ✔ ✔ Pickler Basics ✔ Extensibility ✔ ✔ ✔
73. an extension of pickler combinators that supports subtyping polymorphism; Contributions of the Scala/Pickling project: a framework that enables retrofitting existing types with pickling support and that supports pluggable pickle formats; S r a composable mechanism for static pickler generation; an experimental evaluation that shows that statically-generated pickler combinators can outperform state-of-the-art serialization frameworks and reduce memory usage.
74. qUESTIONS ? heather.miller@epfl.ch @heathercmiller C