Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Beyond JSON: Improving Inter-app Communication

Aaron Quint
September 01, 2015

Beyond JSON: Improving Inter-app Communication

A talk I gave at BaRuCo/Full Stack Fest 2015 about improving inter-app communication by using proctol buffers and non-http protocols.

Aaron Quint

September 01, 2015
Tweet

More Decks by Aaron Quint

Other Decks in Technology

Transcript

  1. Shorthand for Serialization Format, the specific way data is encoded

    and structured for delivery between applications. Format
  2. Break it down • Human Readable/Writable (Well certain humans) •

    Parsers/Clients for every Language • Relatively Fast • Has a few explicit types (String, #, Bool) • Compared to XML/SOAP, Relatively Compact
  3. { "id": "1234", "title": "Beyond JSON”, "author": { "name": “Aaron

    Quint”, "title": “Chief Scientist” }, “favorites": [ { "id": “123”, “user”: “Kat Howard” } ], “created_at": “2015-08-01” }
  4. { "id": "1234", "title": "Beyond JSON”, "author": { "name": “Aaron

    Quint”, "title": “Chief Scientist” “company": { "id": "1334”, "name": “Quirkey NYC” } }, “favorites": [ { "id": “123”, “user”: { “id”: “152”, "name": “Kat Howard”, } } ], “created_at": “2015-08-01” }
  5. What we want • Works well in many languages •

    Relatively Fast • Relatively Compact • Has explicit types • +Handles Change Well
  6. package app; message Meta { required string key = 1;

    required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } app.proto app/*.pb.rb app.pb.go protoc —ruby_out protoc —go_out
  7. image = App::Image.new(:id => "123") bytes = image.encode #=> bytes

    image = App::Image.decode(bytes) image.id #=> 123 package app; message Meta { required string key = 1; required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } image := app.Image{Id: proto.String("123")} bytes, err := proto.Marshal(image) if err != nil { // ... } newImage := &app.Image{} err = proto.Unmarshal(bytes, newImage) if err != nil { // ... } newImage.GetId() #=> "123"
  8. package app; message Meta { required string key = 1;

    required string value = 2; } message Image { optional string id = 1; optional bytes body = 2; optional string extension = 3; optional uint64 size = 4; optional uint32 width = 5; optional uint32 height = 6; optional string filename = 7; repeated Meta meta = 8; } message Request { required string command = 1; required Image image = 2; repeated Meta transform = 3; optional bool returnbody = 4; } message Response { required string status = 1; required string message = 2; optional Image image = 3; } explicit types repeated fields unique tags for backwards and forwards compatibility required fields nested types
  9. ==================== Sizes beefcake: 23b protobuf: 23b json: 71b yajl: 71b

    msgpack: 54b ------------------------------------------------- encode/beefcake 36.915k (± 5.1%) i/s - 184.808k encode/protobuf 34.186k (± 6.7%) i/s - 172.462k encode/json 75.959k (± 8.3%) i/s - 378.367k encode/yajl 81.132k (±11.4%) i/s - 407.199k encode/msgpack 74.240k (±38.1%) i/s - 316.250k Comparison: encode/yajl: 81132.2 i/s encode/json: 75959.2 i/s - 1.07x slower encode/msgpack: 74239.7 i/s - 1.09x slower encode/beefcake: 36915.1 i/s - 2.20x slower encode/protobuf: 34186.1 i/s - 2.37x slower ------------------------------------------------- decode/beefcake 33.048k (± 9.3%) i/s - 164.672k decode/protobuf 45.565k (± 8.7%) i/s - 226.838k decode/json 80.541k (±17.0%) i/s - 390.258k decode/yajl 92.441k (±18.2%) i/s - 442.662k decode/msgpack 126.193k (±12.7%) i/s - 624.618k Comparison: decode/msgpack: 126192.7 i/s decode/yajl: 92441.2 i/s - 1.37x slower decode/json: 80540.7 i/s - 1.57x slower decode/protobuf: 45565.1 i/s - 2.77x slower decode/beefcake: 33047.8 i/s - 3.82x slower https://github.com/quirkey/format_benchmark
  10. Clear wins • Centralized Schema • Easy Serialization into objects

    • Handles removal/addition of fields easily • Real types + nesting • Compact (on the wire)
  11. Downsides • Encoded output is not human readable (does that

    matter?) • Ruby encoding/decoding is not fast (it’s not that slow - wire matters more) • Ruby libraries are not the best (Support for other languages is great, though)
  12. The simplest thing I could possibly think of within the

    bounds of NIH syndrome. TCPEZ http://github.com/paperlesspost/tcpez http://github.com/paperlesspost/tcpez-client
  13. TCPEZ • Client -> Server (Req/Resp) • So far Go

    server + Ruby client • Client is the load balancer • Request and Response encoding/decoding are not part of the protocol. • Each tcpez based project defines its own Request/ Response and these can be in any format • Built in by default: Good logging, Statsd, Protocol Buffers, Pipelined requests!
  14. Clients for all (incl. Ruby) but server is Java only

    now GRPC = A layer above Protobufs over HTTP/2
  15. New version of protobufs has new features: reserved tags, repeated

    compaction, importing, more … GRPC also relies on proto3