Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Unifying Frontend and Backend Development with ...

Unifying Frontend and Backend Development with Scala - Scala Con 2021

Scala can be used for developing both frontend (Scala.js) and backend (Scala JVM) applications. A missing piece has been bridging these two worlds using Scala. We built Airframe RPC, a framework that uses Scala traits as a unified RPC interface between servers and clients. With Airframe RPC, you can build HTTP/1 (Finagle) and HTTP/2 (gRPC) services just by defining Scala traits and case classes. It simplifies web application design as you only need to care about Scala interfaces without using existing web standards like REST, ProtocolBuffers, OpenAPI, etc. Scala.js support of Airframe also enables building interactive Web applications that can dynamically render DOM elements while talking with Scala-based RPC servers. With Airframe RPC, the value of Scala developers will be much higher both for frontend and backend areas.

Presentation Video at Scala Con 2021: https://www.youtube.com/watch?v=MliLBejnyy8

Taro L. Saito

December 03, 2022
Tweet

More Decks by Taro L. Saito

Other Decks in Programming

Transcript

  1. © 2020 Treasure Data 1 Taro L. Saito Treasure Data

    November 2nd, 2021 ScalaCon 2021 Unifying Frontend and Backend Development with Scala
  2. Copyright 1995-2020 Treasure Data. All rights reserved. About Me: Taro

    L. Saito 2 • Treasure Data ◦ Principal software engineer ◦ Building distributed query engine services • OSS activities ◦ Using Scala for 13+ years ▪ Since Scala 2.7 (2008) ◦ Airframe (This talk) ▪ 20+ Scala utility modules ◦ sbt-sonatype ▪ Publishing to Maven central (Sonatype) ◦ etc. Airframe
  3. Copyright 1995-2020 Treasure Data. All rights reserved. Treasure Data: A

    Ready-to-Use Cloud Data Platform 3 Logs Device Data Batch Data PlazmaDB Table Schema Data Collection Cloud Storage Distributed Data Processing Jobs Job Management SQL Editor Scheduler Workflows Machine Learning Treasure Data OSS Third Party OSS Data • Even if you are not an engineer, you can start collecting and analyzing your data
  4. Copyright 1995-2020 Treasure Data. All rights reserved. Even Backend Engineers

    Need Frontend Skills • Creating a better visualization of services ◦ To monitor millions of queries running every day ◦ Created a UI for service monitoring and cluster management with Scala.js 4
  5. Copyright 1995-2020 Treasure Data. All rights reserved. Scala.js Application: Notebook

    Query Editor • Rich SQL editor • Frontend ◦ Scala.js ◦ Monaco editor of Visual Studio Code • Backend ◦ Scala on JVM ◦ RPC server ▪ SQL runner ▪ Task manager • Written only in Scala 5
  6. Copyright 1995-2020 Treasure Data. All rights reserved. Too Many Frontend

    and Backend Frameworks • How do we choose one of the frameworks to use? • It’s almost impossible to find Scala engineers mastering all of these frontend and backend technologies. 6 Web Browsers Scala.js
  7. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC •

    Using plain Scala as an RPC interface between frontend and backend applications • Low-learning cost ◦ Requires only basic Scala knowledge, which can be learned with one Scala book • Our experience so far after introducing Airframe RPC: ◦ A Java engineer in Treasure Data can start writing a Scala RPC service interacting with a Scala.js UI in a week. 7 RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe
  8. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe • GitHub:

    wvlet/airframe • 20+ useful modules written in pure Scala ◦ Cross-built for Scala 2.12, 2.13, and Scala.js ◦ Scala 3 support is in progress ▪ 85%~ completed ▪ WIP: • Tasty reader & Scala reflection compatibility • Core Modules (*: will be explained in this talk) ◦ airframe-log* ◦ airframe-codec* ▪ Object serializer ◦ airframe-http* ▪ HTTP server/client with RPC support ◦ airframe-rx* ▪ Reactive DOM rendering ◦ airframe-json/msgpack ▪ JSON/MsgPack parser ◦ airframe-di ▪ Constructor injection helper ◦ airframe-surface ▪ Compile-time or runtime object inspector ◦ AirSpec: Testing library integrated with Airframe DI ◦ etc. 8 Airframe
  9. Copyright 1995-2020 Treasure Data. All rights reserved. • Various internal

    and third-party Scala/Java libraries were used ◦ e.g., Many web frameworks, utilities, testing frameworks, etc. • The cost of learning individual libraries and frameworks was high Before Airframe: 5 Years Ago (2016) 9 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming Various Libraries Outcome logger launcher object mapper JDBC reader json4s jackson ….
  10. Copyright 1995-2020 Treasure Data. All rights reserved. Current (2021): Consolidating

    the Best Practices into Airframe OSS • Since 2016, we started defining standards in Airframe ◦ Engineers can focus on developing applications instead of learning frameworks 10 Knowledge Experiences Design Decisions Products 24/7 Services Business Values Programming OSS Outcome Airframe
  11. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-log: A Modern

    Logging Library for Scala and Scala.js • There were too many choices just for generating human-readable logs ◦ log4j, slf4j, scala-logging, java.util.logging, twitter/util, etc. • airframe-log ◦ Adding colorful logging methods to your class with LogSupport trait ◦ Showing the source code location 11
  12. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-log usage in

    sbt-sonatype • Shows detailed logs while publishing projects to Maven Central (Sonatype OSS Repository) • News: sbt-sonatype update ◦ Since 2019, sbt-sonatype introduced a new bundle upload process, which can upload thousands of artifact files (jars, source files) to Sonatype within a minute. 12
  13. Copyright 1995-2020 Treasure Data. All rights reserved. • Scala.js compiles

    Scala code into JavaScript, which runs on web browsers • Logging for the developer console of web browsers airframe-log: Debugging Scala.js Code 13
  14. Copyright 1995-2020 Treasure Data. All rights reserved. Frontend Development: Rendering

    DOM with Scala.js • DOM element: ◦ <div class=”container”> …. </div> ▪ tag, attribute, enclosed elements ◦ scalajs-dom library has all necessary functions for manipulating DOM nodes • A previous approach for DOM rendering: ◦ Converting Scala’s XML literal into DOM tree nodes ▪ OSS: monadic-html, Binding.scala ▪ Similar to JSX in Facebook React (XML syntax embedded to JavaScript) ◦ But, XML literal will be deprecated in Scala 3 14
  15. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Functional DOM

    Rendering • The current best practice is defining functions for rendering HTML elements ◦ table, tr, td, body, div, span, etc. • Then, compose these functions • Style 1: Separating DOM attributes and elements into different blocks ◦ ScalaTags, Slinky ◦ Some disadvantages ▪ Too many parentheses! ▪ Formatting with scalafmt is a bit challenging • Style 2: Enclosing both DOM attributes and elements inside a single block ◦ airframe-rx, scalajs-react ◦ More Scala-friendly syntax ◦ The begin/end of a tag are clear 15 ScalaTags airframe-rx
  16. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Rendering Complex

    DOM • RxElement interface: ◦ def render(): RxElement ◦ Support nesting with apply(...) ▪ Embedding Scala functions and collections 16
  17. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Reactive Stream

    Interface Rx[A] • Rx[A] ◦ ReactiveX stream implementation • Rewriting DOM interactively upon upstream data changes ◦ e.g., mouse clicks, getting RPC responses (Future), interval timer events, etc. • Reactive Operators ◦ map, flatMap, filter, join, zip, etc. ◦ The usage is almost the same with Scala collection APIs 17 Airframe
  18. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-rx: Rx.variable •

    Rx.variable ◦ Fire an event if the variable is updated • Interactive DOM Rendering ◦ Subscribing upstream changes ▪ Update DOMs for each change ◦ Cancellation ▪ Unsubscribe the event processing ▪ Necessary for cleaning up event handlers added to the browser • mouse click handler, timer, etc. 18
  19. Copyright 1995-2020 Treasure Data. All rights reserved. RPC: Bridging Scala

    and Scala.js Applications Program Function Call Return Local Function Call (ideal) Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Remote Procedure Call (reality) Network Client Call Return Server Call Return 20 Scala.js Scala.js
  20. Copyright 1995-2020 Treasure Data. All rights reserved. Traditional Approaches for

    Implementing RPC • REST API ◦ Define function interfaces with HTTP endpoints (e.g., GET/POST/PUT/DELETE, etc.) ◦ REST web frameworks for Scala ▪ Play framework, akka-http, Finatra, Finch, skinny-framework, etc. • gRPC ◦ Define function interfaces with Google’s ProtocolBuffers (.proto) schema language ◦ Generate server and client code stubs from proto files ◦ Scala wrappers: ▪ ScalaPB, muScala, akka-grpc, etc. Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return 21
  21. Copyright 1995-2020 Treasure Data. All rights reserved. New Approach: Defining

    RPC Interface with Scala • Scala is a perfect fit for RPC ◦ Scala functions = RPC methods ◦ Scala objects (statically typed) = RPC request/response types • Using the same interface and model classes between servers and clients ◦ No need to think about REST nor ProtocolBuffers for implementing RPC 22 Airframe
  22. Copyright 1995-2020 Treasure Data. All rights reserved. Necessary Building Blocks

    for Implementing RPC • 1. Message serializer and deserializer • 2. Network data format ◦ JSON (REST), Protobuf (gRPC), or MessagePack (Airframe RPC) • 3. RPC interface language ◦ REST API, Protobuf, or Scala • 4. HTTP client and server implementation ◦ Code generator from RPC interface 23 Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Return Server Call Return
  23. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: MessagePack-Based Object

    Serialization • MessagePack ◦ A compact binary format compatible with JSON Object Object Pack Unpack Pack Unpack Server Side Client Side 24 Scala.js JSON
  24. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Predefined Codecs

    • Primitive Codec ◦ ByteCodec, CharCodec, ShortCodec, IntCodec, LongCodec ◦ FloatCodec, DoubleCodec ◦ StringCodec ◦ BooleanCodec ◦ TimeStampCodec • Collection Codec ◦ ArrayCodec, SeqCodec, ListCodec, IndexSeqCodec, MapCodec ◦ OptionCodec, EitherCodec • Java-specific Codec ◦ UUIDCodec, FileCodec, ZonedDateTimeCodec, InstantCodec ◦ JDBCResultSetCodec • etc. 25 Airframe
  25. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Serializing Complex

    Objects Pack Unpack IntCodec StringCodec DoubleCodec MessageCodec.of[A] 26 Serialize Deserialize JSON • airframe-surface ◦ Inspecting object parameters using compile-time macros ◦ Different implementations between Scala 2 and Scala 3
  26. Copyright 1995-2020 Treasure Data. All rights reserved. Object Serialization for

    Multiple Data Formats • Data to Object Mapping ◦ We need many data readers as well as object mappers • A lot of existing libraries for parsing JSON ◦ circe, json4s, play-json, jackson, etc. YAML JDBC ResultSet YAML Parser + Object Mapper Config Object Table Object Object-Relation Mapper JSON JSON Parser + Object Mapper Object 27
  27. Copyright 1995-2020 Treasure Data. All rights reserved. airframe-codec: Using MessagePack

    As A Universal Data Format Object Unpack Pack JDBC ResultSet Pack/Unpack YAML JSON 28 • Once your data are converted into MessagePack, you can use the same object serializer/deserializer (airframe-codec) • Airframe RPC supports JSON format through MessagePack conversion Airframe airframe-codec
  28. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Serializing

    Function Call Request • Serialize function call arguments as Map (arg_name -> arg_value) • An example: ◦ Map(“person” -> Person(1, “leo”), “message” -> “Hello RPC!”) ▪ Serialize this Map into MessagePack with airframe-codec • Mapping to HTTP requests ◦ HTTP method: POST ◦ Path: /(package name).(class name)/(method name) ▪ e.g., /hello.api.v1.MyService/hello ◦ Content body: serialized function call arguments 29
  29. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Implementing

    RPC Servers • Extending the RPC interface trait 30
  30. Copyright 1995-2020 Treasure Data. All rights reserved. Airframe RPC: Backend

    Servers/Clients Are Pluggable RPC Interface RPC Web Server Generates API Documentation 31 Airframe RPC Call RPC Call Implements • Finagle (HTTP/1) • gRPC (HTTP/2) RPC Clients
  31. Copyright 1995-2020 Treasure Data. All rights reserved. • Finagle: Twitter’s

    HTTP server written in Scala ◦ For HTTP/1 services accessible from web browsers running Scala.js code AIrframe RPC: Finagle Backend 32 Airframe Scala.js airframe-rx-html airframe-http-finagle
  32. Copyright 1995-2020 Treasure Data. All rights reserved. • gRPC ◦

    grpc-java (using Netty as backend) • Airframe RPC + gRPC ◦ No .proto file is required ◦ Only Scala Airframe RPC: gRPC Backend 33 Airframe
  33. Copyright 1995-2020 Treasure Data. All rights reserved. RPC Performance Comparison:

    Greeter Service • Airframe RPC ◦ serde: MessagePack (airframe-codec) - Scala case classes ◦ Finagle (HTTP1) or gRPC (HTTP2) • ScalaPB ◦ serde: Protobuf - Scala case classes ◦ gRPC (HTTP2) • grpc-java ◦ serde: Protobuf - Java classes ◦ gRPC (HTTP2) • Notes ◦ Using Finagle with HTTP2 had almost no benefit over HTTP1 ◦ Multiplexing RPC requests into a single connection is the key performance factor ◦ Overhead of ScalaPB over grpc-java ▪ Scala Future (10% overhead) ▪ Mapping Protobuf to Scala case classes (10% overhead) 34 HTTP/1 HTTP/2 (gRPC)
  34. Copyright 1995-2020 Treasure Data. All rights reserved. [Advanced] Extending gRPC

    for Other Data Formats • gRPC is data-format agonistic framework ◦ You can use other than ProtocolBuffers ▪ e.g., JSON, MessagePack • There are two extension points: • MethodDescriptor ◦ Define gRPC endpoints corresponding to Scala functions ◦ Register custom marshallers • Request/ResponseMarshaller ◦ Define how to encode/decode RPC request/response data ◦ We are using airframe-codec here 35
  35. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating RPC

    Clients • sbt-airframe plugin ◦ Read RPC interface classes and generate HTTP client code for the target backend ▪ Using different code generators for Scala, Scala.js, and gRPC sbt-airframe Code Generation RPC Client Scala.js 36 Scala.js Client HTTP/gRPC Client Open API Spec Cross-Language RPC Client
  36. Copyright 1995-2020 Treasure Data. All rights reserved. sbt-airframe: Generating Open

    API Schema • RPC Interface -> Open API schema YAML file • Generating RPC Clients & Swagger Documentation 37 RPC Interface API Documentation Open API Spec (YAML) Cross-Language RPC Client Generate sbt-airframe API Documentation
  37. Copyright 1995-2020 Treasure Data. All rights reserved. Making An RPC

    Call Program Function Serialize Deserialize Request Data Deserialize Response Data Serialize Client Call Response Server Call Response Airframe Scala.js 38
  38. Copyright 1995-2020 Treasure Data. All rights reserved. Summary: Enable Scala-Oriented

    Development with Airframe 39 RPC Interface Scala.js Client RPC Web Server Generates Scala.js Web Application API Documentation RPC Call RPC Call Implements Airframe • Kick start frontend/backend application development with Scala ◦ Only one language to learn ◦ No need to think about REST, ProtocolBuffers, other frontend frameworks at the beginning. • Find example code at https://github.com/wvlet/airframe/ (examples folder)