Upgrade to Pro — share decks privately, control downloads, hide ads and more …

The Future of Streaming: Global Apps, Event Sto...

benstopford
November 12, 2018

The Future of Streaming: Global Apps, Event Stores and Serverless

Stream processing affects a wide range of industries today: capturing sensor data, connecting microservices, processing the workloads of internet giants and giving us a real-time alternative to batch analytics.
While these use cases are exciting and valuable they are only a taste of what is to come. In this talk we look at three areas that are likely to become more prominent: Global Apps, Event Stores and Serverless Stream Processing

benstopford

November 12, 2018
Tweet

More Decks by benstopford

Other Decks in Technology

Transcript

  1. The Future of Streaming: Global Apps, Event Stores and Serverless

    Ben Stopford Office of the CTO, Confluent
  2. THREE TASTES OF THE FUTURE Global Apps: Location independent applications

    Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
  3. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps

    Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Streaming Platform
  4. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
  5. THREE TASTES OF THE FUTURE Global Apps: Location independent applications

    Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
  6. Events change our thinking Monolithic Approach -A database -a variable

    -a singleton -a RPC Event-First Approach - An event - A stream - A log - A stream processor
  7. THREE TASTES OF THE FUTURE Global Apps: Location independent applications

    Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
  8. Events: A fact. An observation of the world. An payment

    A page view A log line A sensor reading
  9. Events come in streams Apps M onitoring Security Apps Apps

    L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M
  10. Traditional systems use mutable state DB This isn’t wrong, it’s

    just lossy Apps Search Mon Apps Apps S T R E A M I N G P L A T F O R M
  11. Events record the user’s journey Shopping Cart Events 2 Trousers

    added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
  12. Stored as a stream Stored statefully (think DB) 12.42 12.44

    12.49 12.50 12.59 Information lost! Event User Journey
  13. 12.42 12.44 12.49 12.50 12.59 We can derive the current

    state (but not the other way around) Apps Apps DERIVE Stream Processor
  14. Streaming is a form of Event Sourcing The current state

    is a projection of the recording Familiar Stateful View LOSSY PROJECTION Stream = Exactly what happened
  15. Formulae 1 – High-Level Architecture • 400 Sensors on car

    • 70,000 derivative measures • Events streamed back to base • Analyzed in real time • Tire modelling • Racing line • Aerodynamics • Machine Learning and Physics Models. • Replayed later for post race analysis. Race Track HQ e.g. Tire modelling: - Temp - Pressure - Suspension compression Stream Processing
  16. Another form of “Event Sourcing” - Record what happened -

    Rewind, replay and rederive (View, App, ML, Physics Model etc.)
  17. New York Times Store of Every article since 1851 (Source

    of Truth) https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
  18. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    KSQL Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing Spark KSQL Orders History 1w All Event stores make data self service (real time & historical)
  19. THREE TASTES OF THE FUTURE Global Apps: Location independent applications

    Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute
  20. Apps Apps Apps Apps Search Monitoring Apps Apps Apps Apps

    Apps Apps Search Monitoring Apps Apps Apps Search NoSQL Apps Apps DWH Hado S T R E A M I N G P L AT F O R M Apps Search NoSQL Apps DWH S T R E A M I N G P L AT F O R M PRODUCER CONSUMER Confluent Cloud 2019 2019
  21. Using FaaS • Write a function • Upload • Configure

    a trigger (HTTP, Event, Object Store, Database, Timer etc.)
  22. FaaS in a Nutshell • Short lived (max ~5 mins)

    • Pay as you use • 0-1000 concurrent functions, autoscales with load • Interesting for spikey compute • Interesting for low priority use cases e.g. CI systems.
  23. Serverless Developer Ecosystem • Runtime diagnostics • Monitoring • Deploy

    loop • Testing • IDE integration Currently quite poor
  24. STREAMING: Event-first - how we think Event-sourced - how we

    store Event-driven - how we combine data and interact
  25. FaaS FaaS FaaS Transaction KSQL Stream processors can act as

    a “data layer” for FaaS ? FaaS FaaS Stateless Stateful (slower elasticity) Orders Payments Customers
  26. FaaS Traditional Application Event-Driven Application Application Database KSQL Stateful Data

    Layer FaaS FaaS FaaS FaaS FaaS Streaming Event-first Event-sourced Event-driven Stateless Stateless Stateless Compute Layer Auto-scaling, correctness, pluggability
  27. THREE TASTES OF THE FUTURE Global Apps: Location independent applications

    Event Stores: Rich recordings of customers and companies Serverless Stream Processing: Melding real-time, elastic data and compute