Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
880
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
58
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
95
Build an Open Source Data Pipeline
ftisiot
0
260
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
640
Fix Your Strings in PostgreSQL
ftisiot
0
230
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
170
Track Database Changes with Apache Kafka
ftisiot
0
240
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
880
Kickstart your Kafka with Faker Data
ftisiot
0
340
Other Decks in Technology
See All in Technology
Oracle Cloud Infrastructureデータベース・クラウド:各バージョンのサポート期間
oracle4engineer
PRO
28
13k
データプロダクトの定義からはじめる、データコントラクト駆動なデータ基盤
chanyou0311
2
320
Terraform CI/CD パイプラインにおける AWS CodeCommit の代替手段
hiyanger
1
240
[CV勉強会@関東 ECCV2024 読み会] オンラインマッピング x トラッキング MapTracker: Tracking with Strided Memory Fusion for Consistent Vector HD Mapping (Chen+, ECCV24)
abemii
0
220
【令和最新版】AWS Direct Connectと愉快なGWたちのおさらい
minorun365
PRO
5
750
iOS/Androidで同じUI体験をネ イティブで作成する際に気をつ けたい落とし穴
fumiyasac0921
1
110
Incident Response Practices: Waroom's Features and Future Challenges
rrreeeyyy
0
160
複雑なState管理からの脱却
sansantech
PRO
1
150
SREが投資するAIOps ~ペアーズにおけるLLM for Developerへの取り組み~
takumiogawa
1
310
ドメイン名の終活について - JPAAWG 7th -
mikit
33
20k
Making your applications cross-environment - OSCG 2024 NA
salaboy
0
190
Terraform Stacks入門 #HashiTalks
msato
0
350
Featured
See All Featured
The Myth of the Modular Monolith - Day 2 Keynote - Rails World 2024
eileencodes
16
2.1k
Git: the NoSQL Database
bkeepers
PRO
427
64k
Into the Great Unknown - MozCon
thekraken
32
1.5k
5 minutes of I Can Smell Your CMS
philhawksworth
202
19k
What's new in Ruby 2.0
geeforr
343
31k
Testing 201, or: Great Expectations
jmmastey
38
7.1k
For a Future-Friendly Web
brad_frost
175
9.4k
The Illustrated Children's Guide to Kubernetes
chrisshort
48
48k
The Cost Of JavaScript in 2023
addyosmani
45
6.8k
ReactJS: Keep Simple. Everything can be a component!
pedronauck
665
120k
Building Flexible Design Systems
yeseniaperezcruz
327
38k
Teambox: Starting and Learning
jrom
133
8.8k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything