Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Reactive Streams: The Glue of the Real-Time Org...

Reactive Streams: The Glue of the Real-Time Organization

The "real-time organization" reacts to information now rather than basing critical decisions on stale data. This requires reimagining how data flows through your organization, moving away from data at rest and moving towards a completely event-based streaming architecture. Events become the lifeblood of the organization, while analytics becomes the brain. Organizations that successfully harness the power of real-time event streaming can transform analytical latencies from hours to seconds.

This session will describe a reference architecture using the Lightbend Reactive Platform and other tools that can be used to build out the real-time organization, leveraging real-time data to produce analytics that result in immediate, actionable insights.

Kevin Webber

March 21, 2016
Tweet

More Decks by Kevin Webber

Other Decks in Programming

Transcript

  1. What to expect » Introduction » A Journey into Reactive

    Streams » Diving into Akka Streams » Code walkthrough and demo » Q&A
  2. Appeal of streams? » Per-event processing (≤ 1 second latencies)

    » Mini-batch processing (≤ 10 second latencies) » Batch processing (≤ 1 hour latencies) » Monitoring, analytics, complex event processing
  3. Reactive Streams? » Reactive Streams is a specification and low-level

    API for library developers » Started as an initiative in late 2013 between engineers at Netflix, Pivotal, and Lightbend
  4. Reactive Streams? » Reactive Streams is a specification and low-level

    API for library developers » Started as an initiative in late 2013 between engineers at Netflix, Pivotal, and Lightbend » Aimed to address two critical challenges » Interoperability between streaming libraries » Flow control over an asynchronous boundary
  5. What is Reactive Streams? » TCK (Technology Compatibility Kit) »

    API (JVM, JavaScript) » Specifications for library developers
  6. Interop » RxJava (Netflix) » Reactor (Pivotal) » Vert.x (RedHat)

    » Akka Streams (Lightbend) » Slick (Lightbend)
  7. Akka Streams » A library to express and run a

    chain of asynchronous processing steps acting on a sequence of elements » DSL for async/non-blocking stream processing » Backpressure enabled by default » Implements the Reactive Streams spec for interoperability » Scala and Java APIs
  8. Let's analyze flight data 1.Read in all flight data from

    2008 2.Transform CSV rows to data structures 3.Compute the average delay per airline 4.Emit the results to console
  9. val g: RunnableGraph[_] = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder => //

    Source val A: Outlet[String] = builder.add(Source.fromIterator(() => flightDelayLines)).out val B: FlowShape[String, FlightEvent] = builder.add(csvToFlightEvent) val C: Inlet[Any] = builder.add(Sink.ignore).in import GraphDSL.Implicits._ // allows us to build our graph using ~> combinators // Graph A ~> B ~> C ClosedShape // defines this as a "closed" graph, not exposing any inlets or outlets }) g.run() // materializes and executes the blueprint
  10. API design Considerations » Immutable, composable stream blueprints » Explicit

    materialization step » No magic at the expense of some extra code
  11. Materialization Separates the what from the how. » Use Source/Flow/Sink

    to create a blueprint » FlowMaterializer turns a blueprint into Akka Actors
  12. val g = RunnableGraph.fromGraph(GraphDSL.create() { implicit builder => import GraphDSL.Implicits._

    // Source val A: Outlet[String] = builder.add(Source.fromIterator(() => flightDelayLines)).out // Flows val B: FlowShape[String, FlightEvent] = builder.add(csvToFlightEvent) val C: FlowShape[FlightEvent, FlightDelayRecord] = builder.add(filterAndConvert) val D: UniformFanOutShape[FlightDelayRecord, FlightDelayRecord] = builder.add(Broadcast[FlightDelayRecord](2)) val F: FlowShape[FlightDelayRecord, (String, Int, Int)] = builder.add(averageCarrierDelay) // Sinks val E: Inlet[Any] = builder.add(Sink.ignore).in val G: Inlet[Any] = builder.add(Sink.foreach(averageSink)).in // Graph A ~> B ~> C ~> D E <~ D G <~ F <~ D ClosedShape }) g.run()