Meetup 2 2019 - Juan Pampliega - Implementing Stream Processing Systems

Transcript

1. Title of class/project By Juan Martín Pampliega Implementing Stream Processing Systems Juan Martín Pampliega
2. Juan Pampliega Information Engineering @ ITBA Co-Founder @ Mutt Data Professor @ ITBA - Especialización en Ciencia de Datos • Working in Data Projects since 2010 at ITBA, Globant (Google), Despegar, Socialmetrix, Jampp, Claro, etc. • Co Founder @ Mutt Data a company specialized in developing projects using Big Data and Data Science. • Developed first production ready stream processing system in 2015 @juanpampliega | juan@muttdata.ai
3. Contents Why Stream Processing? Fundamental Concepts Technology Landscape Summary Implementing Stream Processing Systems 01 02 03 04
4. 01 Why Stream Processing ?
5. Event-Command vs Event-Driven For organizations to become real time, they must become event driven In Event-Command pattern (like REST applications) the endpoint is known, the method being called is also known and lastly the calls tended to return a value. In Event-Driven pattern services communicate only by generating events that can be reused by any service in the system which leads to less coupling.
6. Service Oriented
7. Event Driven: Unified Log
8. Event Streams Data is generally born as continuous event streams Batch processing => bounded datasets Stream processing => unbounded datasets Stream processing means computing on data directly as it is produced or received.
9. Batch Processing Traditional data processing storing streams of data in databases and later processing it using ETLs. Still needed when you need to explore data an when you are not yet sure what you want to do with it.
10. Stream Processing In stream processing the application logic, analytics, and queries exist continuously, and data flows through them continuously.
11. To produce results in real time a system must continuously compute and update results with each new event. Modern applications and microservices should operate in an event-driven fashion. Their logic and computation is triggered by events. Unified event-driven applications and real-time analytics
12. 02 Fundamental Concepts
13. Operations over Streams FILTER JOIN
14. Stateful Streaming Processing Stateful stream processing is a subset of stream processing in which the computation maintains contextual state. This state is used to store information derived from the previously-seen events.
15. Windows Partition a stream of data into discrete batches. Needed to compute metrics that require context (average by minute, partial counts, etc.)
16. Event Time vs Processing Time • Processing and transport technologies. • Characteristics of the data source. (distribution, throughput, burstiness, etc.) • Hardware. (network, RAM, CPUs, etc.) • FAILURES & RETRIES
17. Event Time vs Processing Time
18. Operations over Streams PROCESSING TIME EVENT TIME
19. Streaming State and Windows
20. Fault Tolerant Local State Local state is a fundamental primitive in stream processing. It can be indexed and accessed in a variety of rich ways. Local, in-process data access is much faster. It’s easier to isolate. Implemented with in memory hash table, bloom filters, bit maps, RocksDB like systems, etc.
21. Critical Questions in Stream Processing Tyler Akidau (tech lead for internal streaming data processing systems @ Google) defined 4 critical questions any stream processing system should be able to answer: What results are calculated? Where in event time are results calculated? When in processing time are results materialized? How do refinements of results relate?
22. Watermarks and Triggers
23. 03 Technology Landscape
24. Ververica Platform
25. Apache Flink
26. Companies that use Apache Flink
27. Apache Beam Beam Model: Fn Runners Apache Flink Apache Spark Beam Model: Pipeline Construction Other Languages Beam Java Beam Python Execution Execution Cloud Dataflow Execution 1. The Beam Model: What / Where / When / How 2. SDKs for writing Beam pipelines -- starting with Java 3. Runners for Existing Distributed Processing Backends ◦ Apache Apex ◦ Apache Flink ◦ Apache Spark ◦ Google Cloud Dataflow ◦ Local (in-process) runner for testing
28. Google Cloud Dataflow
29. Spark Streaming Platform
30. Confluent Platform
31. 04 Resources
32. Ben Stopford, The Data Dichotomy: Rethinking the Way We Treat Data and Services https://www.confluent.io/blog/data-dichotomy-rethinking-the-way-we-treat-data-and- services/ Tyler Akidau, Streaming 101: The world beyond batch https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-101 Tyler Akidau, Streaming 102: The world beyond batch https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102 Ververica, What is Stream Processing? https://www.ververica.com/what-is-stream-processing Apache Flink Documentation https://ci.apache.org/projects/flink/flink-docs-release-1.8/ Apache Beam https://beam.apache.org/
33. Title of class/project By Juan Martín Pampliega Thank you ! Implementing Stream Processing Systems By Juan Martín Pampliega We are hiring Data Engineers! info@muttdata.ai