Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
900
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
71
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
100
Build an Open Source Data Pipeline
ftisiot
0
270
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
670
Fix Your Strings in PostgreSQL
ftisiot
0
240
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
180
Track Database Changes with Apache Kafka
ftisiot
0
250
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
900
Kickstart your Kafka with Faker Data
ftisiot
0
360
Other Decks in Technology
See All in Technology
2024.02.19 W&B AIエージェントLT会 / AIエージェントが業務を代行するための計画と実行 / Algomatic 宮脇
smiyawaki0820
14
3.8k
エンジニアのためのドキュメント力基礎講座〜構造化思考から始めよう〜(2025/02/15jbug広島#15発表資料)
yasuoyasuo
18
7k
Windows の新しい管理者保護モード
murachiakira
0
120
分解して理解する Aspire
nenonaninu
1
370
速くて安いWebサイトを作る
nishiharatsubasa
13
15k
クラウドサービス事業者におけるOSS
tagomoris
3
920
偶然 × 行動で人生の可能性を広げよう / Serendipity × Action: Discover Your Possibilities
ar_tama
1
1.3k
プロダクトエンジニア 360°フィードバックを実施した話
hacomono
PRO
0
110
Visualize, Visualize, Visualize and rclone
tomoaki0705
4
2k
エンジニアが加速させるプロダクトディスカバリー 〜最速で価値ある機能を見つける方法〜 / product discovery accelerated by engineers
rince
4
430
レビューを増やしつつ 高評価維持するテクニック
tsuzuki817
1
780
室長と気ままに学ぶマイクロソフトのビジネスアプリケーションとビジネスプロセス
ryoheig0405
0
370
Featured
See All Featured
The Invisible Side of Design
smashingmag
299
50k
How to train your dragon (web standard)
notwaldorf
91
5.8k
Designing for Performance
lara
604
68k
Learning to Love Humans: Emotional Interface Design
aarron
273
40k
Why Our Code Smells
bkeepers
PRO
336
57k
Unsuck your backbone
ammeep
669
57k
For a Future-Friendly Web
brad_frost
176
9.5k
Become a Pro
speakerdeck
PRO
26
5.1k
[RailsConf 2023] Rails as a piece of cake
palkan
53
5.3k
Git: the NoSQL Database
bkeepers
PRO
427
64k
The Language of Interfaces
destraynor
156
24k
RailsConf 2023
tenderlove
29
1k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything