Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
940
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
95
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
120
Build an Open Source Data Pipeline
ftisiot
0
290
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
700
Fix Your Strings in PostgreSQL
ftisiot
0
260
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
210
Track Database Changes with Apache Kafka
ftisiot
0
270
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
940
Kickstart your Kafka with Faker Data
ftisiot
0
400
Other Decks in Technology
See All in Technology
Telemetry APIから学ぶGoogle Cloud ObservabilityとOpenTelemetryの現在 / getting-started-telemetry-api-with-google-cloud
k6s4i53rx
0
150
Kiro と Q Dev で 同じゲームを作らせてみた
r3_yamauchi
PRO
1
100
OPENLOGI Company Profile for engineer
hr01
1
38k
AI時代の大規模データ活用とセキュリティ戦略
ken5scal
0
160
Amazon Qで2Dゲームを作成してみた
siromi
0
150
ロールが細分化された組織でSREと協働するインフラエンジニアは何をするか? / SRE Lounge #18
kossykinto
0
220
S3 Glacier のデータを Athena からクエリしようとしたらどうなるのか/try-to-query-s3-glacier-from-athena
emiki
0
230
AWS DDoS攻撃防御の最前線
ryutakondo
1
170
LLM 機能を支える Langfuse / ClickHouse のサーバレス化
yuu26
9
2.4k
リモートワークで心掛けていること 〜AI活用編〜
naoki85
0
180
Amazon Q Developerを活用したアーキテクチャのリファクタリング
k1nakayama
2
220
Amazon Inspector コードセキュリティで手軽に実現するシフトレフト
maimyyym
0
120
Featured
See All Featured
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
31
2.2k
Navigating Team Friction
lara
188
15k
Adopting Sorbet at Scale
ufuk
77
9.5k
JavaScript: Past, Present, and Future - NDC Porto 2020
reverentgeek
50
5.5k
VelocityConf: Rendering Performance Case Studies
addyosmani
332
24k
GraphQLとの向き合い方2022年版
quramy
49
14k
Imperfection Machines: The Place of Print at Facebook
scottboms
268
13k
GraphQLの誤解/rethinking-graphql
sonatard
71
11k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
30
9.6k
Become a Pro
speakerdeck
PRO
29
5.5k
Side Projects
sachag
455
43k
The Straight Up "How To Draw Better" Workshop
denniskardys
235
140k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything