Event Streaming and the Future of Applications

Events fundamentally change the way we build systems. As a means of communication they loosely couple microservices together, meaning sections of an architecture become isolated from the failure semantics of the rest. As a data model they tie communication and storage together via a technique called event sourcing, which retains information exactly as it happened, primed for future use cases or for analytics. Events also imply motion: Data moves from one application to another, with stream processors used to manipulate these data flows as they progress. Finally, events are the backbone of serverless architectures. A programming model that is likely to be prevalent in our future.
In this talk, we’ll look at the relationship between these different aspects of the humble event, providing insight into why the paradigm is important now, as well as reflecting on its inevitable positioning in our future.

Transcript

1. Event Streaming and the Future of Applications Ben Stopford Office of the CTO, Confluent
2. Old and emerging categories Relational Database Data warehouse Messaging Complex Event Processing NoSQL & Big Data Event Streaming 1970 1980 1990 2000 2010 2020 Cloud & Serverless
3. Event Storage Kafka stores petabytes of data Stream Processing Real-time processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Roots in big data messaging
4. Events: A fact. An observation about the world, captured at a point in time. A payment An order A page view A log line A sensor reading
5. Events come in streams Apps M onitoring Security Apps Apps L A T F O R M Event Stream Order of events is important Apps M onitoring Apps Apps O R M Older Newer
6. Streams let us react to the world as it changes over time Order Created Order Confirmed Order Paid Order Shipped
7. Stream Processors React, Combine, Filter & Summarize Event Streams KSQL
8. If you embrace events it changes your programs in two main ways: 1. The way you store data 2. The way your programs communicate
9. Let’s start with a simple example DB What’s wrong with this architecture? Apps Search Apps Apps
10. Events create a timeseries of things a user did Shopping Cart Events 2 Trousers added 1 Jumper added 1 Trousers removed 1 Hat added Checkout Shopping Cart Event User Journey 12.42 12.44 12.49 12.50 12.59
11. Stored as a stream Stored as “Current State” (i.e. CRUD) 12.42 12.44 12.49 12.50 12.59 Information lost! Event User Journey
12. 12.42 12.44 12.49 12.50 12.59 We can derive the ‘current state view’ (but not the other way around) Apps Apps DERIVE Stream Processor
13. Events have 3 main benefits as a storage model
14. Business Lens Evidentiary Accountants don’t use erasers (e.g. audit, ledger, git)
15. Ops Lens Replayability (e.g. recovering data after programmatic error)
16. Analytics Lens • Trends and behaviors • (e.g. insight, metrics, ML)
17. What do such systems look like?
18. New York Times – Event Based Storage in Kafka. (“View” continuously derived from the log) Source of Truth Every article since 1851 https://www.confluent.io/blog/publishing-apache-kafka-new-york-times/ Normalized assets (images, articles, bylines, tags all separate messages) Denormalized into “Content View”
19. If you embrace events it changes your programs in two main ways: 1. The way you store data 2. The way your programs communicate
20. Increasing Complexity Apps Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
21. Apps Search NoSQL Mo Se Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Architectures grow Loosely coupled
22. Apps Search NoSQL Mo Se Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Ap Apps Apps Apps App Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Apps Apps Apps Apps Apps Apps App Event Preparation • Join • Filter • Transform • Enrich (from tables) • Summarize
23. Make the consumption of real-time event streams easy, even if the data comes from many different places.
24. KSQL is a kind of Database for Creating new Event Streams Orders Service Payment Service Customer Service KSQL Aggregate Events Apps Search Ap Order Payment Customer SELECT * FROM…
25. Serverless functions, being inherently event-driven, are a great example
26. What is a Serverless Function? • Write a function • Upload • Configure an event trigger (HTTP, Event, Object Store, Database, Timer etc.) • Fully managed (Runs in a container pool)
27. The benefits of Serverless Functions stem from being event-driven • Auto-scales with load ~ 0-1000 concurrent functions • Event backlog drives autoscaling • Pay for execution time not resources used
28. Harder than current approaches Easier than current approaches Amazon Google Microsoft Serverless programming will likely become prevalent
29. Serverless Functions are event-driven They aren’t “streaming”
30. Default: Complex, Timing issues, Scaling limits Customers Event Source FaaS FaaS FaaS Orders Event Source Payments Event Source
31. FaaS FaaS FaaS KSQL Stream processors can act as a “data layer” for FaaS ? FaaS FaaS High elasticity Stateful operations Orders Payments Customers All data is in the “enriched” event
32. Summary • Events underpin the storage models of truthful/factful systems. • Events decouple architectures and provide real-time data. • Serverless is infrastructure made event-driven. • Stream Processing is the natural corollary, for data.
33. Thank you @benstopford Book: https://www.confluent.io/designing-event-driven-systems Signing at Confluent booth immediately after this talk