Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Trends in Event Streaming

Avatar for benstopford benstopford
February 03, 2020

Trends in Event Streaming

A look at a number of key trends in Event Streaming for 2020 including:
- Batch to realtime
- Contextual Event-Driven applications
- Business processes becoming more realtime
- More copies of data, less deviation
- Self-service data
- Migration to cloud

Avatar for benstopford

benstopford

February 03, 2020
Tweet

More Decks by benstopford

Other Decks in Technology

Transcript

  1. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + An Event Streaming Platform is a Data Platform for Data in Motion
  2. Data in one place vs data in many places Apps

    Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps h L Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App App Apps Search Monitoring Apps Apps App Apps Search Monitoring Apps Apps Apps h Monitoring Apps Apps App App Data is accurate. Very tightly coupled. Easy for different apps to evolve independently How do I join? Data gets out of sync
  3. Three options • Single database: all data in one place

    • Doesn’t scale in people terms • Microservices: Data spread in many “golden sources” • Opperationanal issues ensue • Data get’s copied everywhere as teams attempt to hit deadlines • Event Streaming • Single source of truth • Applications take a copy, that can be re-sourced when necessary
  4. Apps Monitoring Security Apps Apps S T R E A

    M I N G P L AT F O R M Apps Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Monitoring Security Apps Apps S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Mon Sec Apps Apps S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL Apps App S T R E A M I N G P L AT F O R M Apps Search NoSQL Monitoring Security Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T R E A M I N Apps Search NoSQL Apps DWH S T R E A M I N G P L AT App Apps Search NoSQL Apps S T R E A M I N G P L AT Apps Search NoSQL Apps Apps DWH Hadoop S T R E A M I N G P L AT F O R M App Apps Search NoSQL S T Apps Search NoSQL DWH S T R E App Apps Apps Search Apps App Apps Apps Apps Search Apps Apps App Apps Apps Search App Kafka: Source of truth Evolution of software systems Monolith Distributed Monolith Microservices Event-Driven Microservices
  5. Event Storage Kafka stores petabytes of data Stream Processing Real-time

    processing over streams and tables Scalability Clusters of hundreds of machines. Global. + + + Event Streaming
  6. Trade Surveillance Project • 9 months sourcing 16 data sets

    • Different formats (including for historical extracts) • Batch based approach
  7. Event Streams Orders Payments Customers Distinct Visits Destination Spark Postgres

    Lambda Other Kafka Select Organizational Events Stream Processing SELECT * FROM ORDERS O, CUSTOMERS C WHERE O.REGION = ‘EU’ AND C.TYPE = ‘Platinum’ Msgs/Day Customers Stream Processing DB App Orders History 1w All Making Data Self Service
  8. Confluent Cloud • Dramatically cheaper than self managed for most

    use cases • No operational burden • No need to size clusters or pre- purchase hardware • Scale from zero (i.e. free) to whatever you need.
  9. Trends • Batch to real-time • Contextual event driven applications

    • Increasingly automated business processes • More copies, less deviation • Self-service data • Migration to cloud