Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
930
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
94
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
120
Build an Open Source Data Pipeline
ftisiot
0
290
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
690
Fix Your Strings in PostgreSQL
ftisiot
0
250
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
200
Track Database Changes with Apache Kafka
ftisiot
0
260
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
930
Kickstart your Kafka with Faker Data
ftisiot
0
390
Other Decks in Technology
See All in Technology
CI/CD/IaC 久々に0から環境を作ったらこうなりました
kaz29
1
190
ドメイン特化なCLIPモデルとデータセットの紹介
tattaka
1
210
250627 関西Ruby会議08 前夜祭 RejectKaigi「DJ on Ruby Ver.0.1」
msykd
PRO
2
340
KubeCon + CloudNativeCon Japan 2025 Recap Opening & Choose Your Own Adventureシリーズまとめ
mmmatsuda
0
180
AI導入の理想と現実~コストと浸透〜
oprstchn
0
110
25分で解説する「最小権限の原則」を実現するための AWS「ポリシー」大全 / 20250625-aws-summit-aws-policy
opelab
9
1.2k
Geminiとv0による高速プロトタイピング
shinya337
0
110
Node-RED × MCP 勉強会 vol.1
1ftseabass
PRO
0
170
AWS Organizations 新機能!マルチパーティ承認の紹介
yhana
1
170
AIとともに進化するエンジニアリング / Engineering-Evolving-with-AI_final.pdf
lycorptech_jp
PRO
0
120
「Chatwork」の認証基盤の移行とログ活用によるプロダクト改善
kubell_hr
1
210
How Community Opened Global Doors
hiroramos4
PRO
1
130
Featured
See All Featured
Making Projects Easy
brettharned
116
6.3k
Code Reviewing Like a Champion
maltzj
524
40k
[RailsConf 2023] Rails as a piece of cake
palkan
55
5.6k
Performance Is Good for Brains [We Love Speed 2024]
tammyeverts
10
940
Building Better People: How to give real-time feedback that sticks.
wjessup
367
19k
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Easily Structure & Communicate Ideas using Wireframe
afnizarnur
194
16k
Navigating Team Friction
lara
187
15k
A designer walks into a library…
pauljervisheath
207
24k
The Cult of Friendly URLs
andyhume
79
6.5k
GraphQLとの向き合い方2022年版
quramy
49
14k
Cheating the UX When There Is Nothing More to Optimize - PixelPioneers
stephaniewalter
281
13k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything