Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Change Stream Processing with Apache Flink and Decodable

Change Stream Processing with Apache Flink and Decodable

Change Data Capture (CDC) has earned its popularity in the stream processing world with the rise of Debezium. Apache Flink supports stream processing over change logs natively.

Decodable leverages both open-source projects and offers a service that simplifies CDC processing with rich APIs and SQL. In this webinar, Sharon and Gunnar will demonstrate some use cases around real-time change stream processing with Decodable, including stream joins from multiple sources, detecting critical change events, and more.

Bring all your questions around CDC and stream processing, so we can make this interactive and fun for everyone!

Gunnar Morling

November 17, 2022

More Decks by Gunnar Morling

Other Decks in Programming


  1. Today’s Objectives: Learn about… 01. CDC What’s change data capture

    and why should I care? 02. CDC + Flink What do I gain by integrating change data capture with Apache Flink? 03. Demo Enough of the talk, show me the code!
  2. @sharon_rxie @gunnarmorling Your Presenters Sharon Xie Gunnar Morling • Founding

    engineer of Decodable • 5 years of Flink experience • @sharon_rxie • Open source software engineer • Former lead of Debezium • @gunnarmorling
  3. @sharon_rxie @gunnarmorling Change Streams Event Format • Encode insert/update/delete operations

    to a record • Keyed by primary key • Contain both before and after versions key: { user_id: 13 } # ← primary key value: { op: ‘u’, # ← i = insert, u = update, d = delete before: { status: ‘active’, ... }, after: { status: ‘inactive’, ... }, ... }
  4. Questions • How to integrate data sinks? • How to

    stream CDC events to consumers? • How to process events?
  5. @sharon_rxie @gunnarmorling Apache Flink Stream Processing of Change Data Events

  6. Questions • Do I need to program in Java? •

    Do I need to set up and operate Flink? • How to gain operational insight into my data flows?
  7. @sharon_rxie @gunnarmorling Standing on the Shoulders of Giants Upstream Open-Source

    Communities https://debezium.io/ https://flink.apache.org/ https://ververica.github.io/flink-cdc-connectors/ Flink CDC Debezium Apache Flink Flink CDC Connectors
  8. @sharon_rxie @gunnarmorling CDC allows data to be independently processed Apache

    Flink natively supports to process CDC events Decodable is built on top of Flink and offers change stream processing as a service Wrap Up
  9. decodable.co 2022 Thanks! decodable.co @sharon_rxie | @gunnarmorling | @decodableco {

    sharon | gunnar } @ decodable.co Free trial Twitter E-Mail
  10. @sharon_rxie @gunnarmorling Getting Started with Decodable • Visit http://decodable.co •

    Start Free http://app.decodable.co • Read the docs http://docs.decodable.co • Check out the Decodable YouTube Channel • Join our community Slack channel • Call [email protected]