Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Track Database Changes with Apache Kafka

FTisiot
September 11, 2021
Technology
0
240

Track Database Changes with Apache Kafka

Tired of the spaghetti architecture created by hundreds applications and ETL flows dependencies making every database table change a matter of 100 people meeting? Understand how you can decouple and integrate better your systems with Apache Kafka and Kafka Connect!

FTisiot

September 11, 2021
Tweet

More Decks by FTisiot

See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
60
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
95
Build an Open Source Data Pipeline
ftisiot
0
260
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
650
Fix Your Strings in PostgreSQL
ftisiot
0
230
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
180
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
890
Kickstart your Kafka with Faker Data
ftisiot
0
350
Event-Driven Applications: Apache Kafka and Python
ftisiot
0
890

Other Decks in Technology

See All in Technology
Storage Browser for Amazon S3
miu_crescent
1
140
PHP ユーザのための OpenTelemetry 入門 / phpcon2024-opentelemetry
shin1x1
1
200
サイバー攻撃を想定したセキュリティガイドライン 策定とASM及びCNAPPの活用方法
syoshie
3
1.2k
re:Invent をおうちで楽しんでみた ～CloudWatch のオブザーバビリティ機能がスゴい！/ Enjoyed AWS re:Invent from Home and CloudWatch Observability Feature is Amazing!
yuj1osm
0
120
KnowledgeBaseDocuments APIでベクトルインデックス管理を自動化する
iidaxs
1
260
20241214_WACATE2024冬_テスト設計技法をチョット俯瞰してみよう
kzsuzuki
3
450
宇宙ベンチャーにおける最近の情シス取り組みについて
axelmizu
0
110
Microsoft Azure全冠になってみた ~アレを使い倒した者が試験を制す！？~/Obtained all Microsoft Azure certifications Those who use "that" to the full will win the exam! ?
yuj1osm
2
110
Snykで始めるセキュリティ担当者とSREと開発者が楽になる脆弱性対応 / Getting started with Snyk Vulnerability Response
yamaguchitk333
2
180
KubeCon NA 2024 Recap: How to Move from Ingress to Gateway API with Minimal Hassle
ysakotch
0
200
podman_update_2024-12
orimanabu
1
270
オプトインカメラ：UWB測位を応用したオプトイン型のカメラ計測
matthewlujp
0
170

Featured

See All Featured
StorybookのUI Testing Handbookを読んだ
zakiyama
27
5.3k
Reflections from 52 weeks, 52 projects
jeffersonlam
347
20k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
365
25k
Build your cross-platform service in a week with App Engine
jlugia
229
18k
Building Better People: How to give real-time feedback that sticks.
wjessup
365
19k
BBQ
matthewcrist
85
9.4k
Stop Working from a Prison Cell
hatefulcrawdad
267
20k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5k
Raft: Consensus for Rubyists
vanstee
137
6.7k
Speed Design
sergeychernyshev
25
670
A Tale of Four Properties
chriscoyier
157
23k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
26
1.5k

Transcript

  1. I don’t Want to Miss a Thing Track Database Changes

    with Apache Kafka Francesco Tisiot - Developer Advocate @ftisiot

  2. @ftisiot

  3. @ftisiot

  4. @ftisiot

  5. @ftisiot

  6. @ftisiot What is Apache Kafka?

  7. @ftisiot Topic A Topic B 0 1 2 3 4

    0 1 2 3 Producer Consumer Producer Consumer Consumer

  8. @ftisiot Events in a Database?

  9. @ftisiot Table Log vs The Fridge Dilemma

  10. @ftisiot Table Log 5 3 6 +7 +3 +6 -5

    -2 +5

  11. @ftisiot Integrating Apache Kafka

  12. @ftisiot Kafka Connect Source Kafka Connect Sink

  13. @ftisiot Changes in Kafka?

  14. @ftisiot Query Based Approach

  15. @ftisiot

  16. @ftisiot Log Based Approach

  17. @ftisiot

  18. @ftisiot Debezium

  19. @ftisiot

  20. @ftisiot Query Bulk Query Timestamp Log INSERT UPDATE DELETE NOTES

  21. @ftisiot Resources https://aiven.io https://debezium.io/ https://aiven.io/blog/using-kafka-connect-jdbc-source- a-postgresql-example https://kafka.apache.org/