Timing is Everything: Understanding Event-Time Processing in Flink SQL (Current 24)

In the stream processing context, event-time processing means the events are processed based on when the events occurred, rather than when the events are observed (processing-time) in the system. Apache Flink has a powerful framework for event-time processing, which plays a pivotal role in ensuring temporal order and result accuracy.

In this talk, we will introduce Flink event-time semantics and demonstrate how watermarks as a means of handling late-arriving events are generated, propagated, and triggered using Flink SQL. We will explore operators such as window and join that are often used with event time processing, and how different configurations can impact the processing speed, cost and correctness.

Join us for this exploration where event-time theory meets practical SQL implementation, providing you with the tools to make informed decisions for making optimal trade-offs.

Transcript

1. Timing is Everything Understanding Event-Time Processing in Flink SQL Sharon Xie，Head of Product Founding Engineer @ Decodable
2. Agenda - Flinkʼs event time processing model - Use cases in Flink SQL
3. What is Apache Flink • Highly scalable stream processing engine • Exactly-once processing semantics • Layered APIs: SQL (easy to use) ↔ Java/Python DataStream (expressive) • Support event-driven applications, streaming ETL pipelines, streaming analytics
4. Context • Flink streaming mode • Flink SQL
5. Event Immutable record containing the detail of something that happened at some point in time.
6. Time in Flink Event Time • The time at which the event happened Processing Time • The time at which the event is observed by Flink
7. When is event time used? • Decisions or insights based on when the event happens ◦ Monitoring and alerting ◦ Time-based compute or analytics
8. Event time vs Processing Time • Event time is < processing time • The lag is arbitrary • Events can be out-of-order
9. Challenges How do you know when all of the events are received for a particular window?
10. Watermark • Measures the progress of event time • Tracks the maximum event time seen • Indicates the completeness of the event time
12. Watermark
13. Create table sensors ( id bigint, `value` integer _time timestamp(3), watermark for _time as _time - interval '3' minutes ) WITH ( 'scan.watermark.emit.strategy'='on-event', ... ); Define Watermark
14. Watermark Generation (on-event)
15. There is a window that ends at 105. When can the window close? Quiz
16. There is a window that ends at 105. When can the window close? Quiz - Answer
17. Multiple sources/partitions
18. Idle source/partition • If a partition is idle (no events), the watermark will not advance • No result will be produced • Solutions ◦ Configure source idle timeout ▪ set table.exec.source.idle-timeout = 1m ◦ Balance the partitions
19. Implications • Tradeoff between Correctness and Latency • Latency ◦ Results of a window is only seen after the window closes • Correctness ◦ Late arriving events are discarded after the window is closed
20. Correctness VS Latency In general: Alerting and monitoring: latency Timely analytics: correctness
21. But…can I have both? • Yes! Flink can process & emit “updatesˮ (changelog) • No watermark is needed • Downstream system must support “updatesˮ • Itʼs costly - need to store global state
22. Trade-offs
23. Quick Summary • Event time is the time when the event happens • Flink uses watermark to account for out-of-order events • Watermark strategy allows trade-off between accuracy and latency
24. Event-time Usage In Flink SQL • Windowed Aggregations • Window join • Temporal join
25. Windowing Put unbounded events into finite-sized temporal buckets, over which computation is applied.
26. Window Types • Tumble / Fixed • Hop / Sliding • Cumulative • Session
27. Window Types - Tumble/Fixed Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/ • Fixed window size • No overlapping • Each event belongs to exactly 1 window
28. Flink SQL Window TVF • TVF  TableValued Function • Returns a new relation with all columns of original stream and additional 3 columns: ◦ window_start, window_end, window_time
29. Tumble Window TVF
30. 1st window_start value Easy calculation: The nearest multiple of window size before 1st event time Formula: floor( 1st event time - reference time*) / window size ) * window size + window_offsets * reference time is 19700101T000000.000 https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#wi ndow-offset
31. Quiz Whatʼs the fixed window start value for the event time 101 with window size of 5min?
32. Quiz - Answer Whatʼs the fixed window start value for the event time 101 with window size of 5min? • 100
33. Example - Tumble Window
34. Window Types - Hop/Slide Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/ • Fixed window size • Overlaps when slide < window size
35. Window Types - Cumulative Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/window-tvf/ • Similar to tumble window, but with early firing at the defined interval • Defined by max window size and window step
36. Window Types - Session A new window is created when the consecutive event time > session gap Ref: https://nightlies.apache.org/flink/flink-docs-release-1.19/docs/dev/table/sql/queries/window-tvf/#session
37. Window Join • A window join adds the dimension of time into the join criteria themselves. • Use case: compute click-through events
38. Example - hop window join
39. Example - hop window join
40. Example - hop window join
41. Temporal Join • Enrich a stream with the value of the joined record at the event time. • Example: Continuously computing the price for each order based on the exchange rate happened when the order is placed
42. Example - temporal join Ref: https://nightlies.apache.org/flink/flink-docs-release-1.18/docs/dev/table/sql/queries/joins/#temporal-joins
45. Summary • Event time is essential for timely response and analytics • Watermark and windowing are the key concepts • Flink SQL simplifies event time processing
46. Thank you Q&A @sharon_rxie
47. Decodable Talks at Current 2024 Timing is Everything: Understanding Event-Time Processing in Flink SQL 🗣 Sharon Xie 📆 Tuesday 4pm 🗺 Ballroom F Data Contracts In Practice With Debezium and Apache Flink 🗣 Gunnar Morling 📆 Tuesday 3pm 🗺 Meeting Room 18C So You Want to Write a User-Defined Function (UDF) for Flink? 🗣 Hans-Peter Grahsl 📆 Wednesday 1:30pm 🗺 Ballroom F The Joy of JARs (and Other Flink SQL Troubleshooting Tales) 🗣 Robin Moffatt 📆 Wednesday 3pm 🗺 Ballroom F