Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Apache Kafka JDBC Source Connector: What could ...

FTisiot
April 26, 2022

Apache Kafka JDBC Source Connector: What could go wrong?

Slides of my #KafkaSummit talk "Apache Kafka JDBC Source Connector: What could go wrong?"

When needing to source Database events into Apache Kafka, the JDBC source connector usually represents the first choice for its flexibility and the almost-zero setup required on the database side. But sometimes simplicity comes at the cost of accuracy and missing events can have catastrophic impacts on our data pipelines.

In this session we'll understand how the JDBC source connector works and explore the various modes it can operate to load data in a bulk or incremental manner. Having covered the basics, we'll analyse the edge cases causing things to go wrong like infrequent snapshot times, out of order events, non-incremental sequences or hard deletes.

Finally we'll look at other approaches, like the Debezium source connector, and demonstrate how some more configuration on the database side helps avoid problems and sets up a reliable source of events for our streaming pipeline.

Want to reliably take your Database events into Apache Kafka? This session is for you!

FTisiot

April 26, 2022
Tweet

More Decks by FTisiot

Other Decks in Technology

Transcript

  1. @ftisiot | @aiven_io Common Challenges Data Types Out of Memory

    Errors Number Mapping numeric.mapping defaultRowFetchSize
  2. @ftisiot | @aiven_io 1 2 3 Id Name 1 2

    3 Incremental = No Updates!
  3. @ftisiot | @aiven_io JDBC Limits Polling Time Out of Order

    Events Load on the DB Updates/Deletions Require Extra Fields
  4. @ftisiot | @aiven_io JDBC Limits Polling Time Out of Order

    Events Load on the DB Updates/Deletions Require Extra Fields All Events Near Real Time Tracked as per Log Minimal Load No Extra Fields
  5. @ftisiot | @aiven_io https:/ /aiven.io Debezium Connector JDBC Source Connector

    in Action Debezium Connector in Action JDBC Connector https:/ /ftisiot.net/talks/kafka-jdbc-what-can-go-wrong/ kafka-summit-2022 500$