Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
0
860
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Tweet
Share
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
53
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
93
Build an Open Source Data Pipeline
ftisiot
0
260
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
610
Fix Your Strings in PostgreSQL
ftisiot
0
230
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
170
Track Database Changes with Apache Kafka
ftisiot
0
230
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
860
Kickstart your Kafka with Faker Data
ftisiot
0
320
Other Decks in Technology
See All in Technology
自作Cコンパイラ 8時間の奮闘
soukouki
0
840
アプリをリリースできる状態に保ったまま 段階的にリファクタリングするための 戦略と戦術 / Strategies and tactics for incremental refactoring
yanzm
6
1.4k
Fediverse Discovery Providers overview
andypiper
0
170
AIを活用した柔軟かつ効率的な社内リソース検索への取り組み
cygames
0
180
React Aria で実現する次世代のアクセシビリティ
ryo_manba
4
1.2k
突撃! 隣のAmazon Bedrockユーザー 〜YouはどうしてAWSで?〜
minorun365
PRO
3
390
技術的負債解消の取り組みと専門チームのお話
bengo4com
0
330
『GRANBLUE FANTASY Relink』キャラクターの魅力を支えるリグ・シミュレーション制作事例
cygames
0
120
「家族アルバム みてね」における運用管理・ オブザーバビリティの全貌 / Overview of Operation Management and Observability in FamilyAlbum
isaoshimizu
4
160
Developer Experienceを向上させる基盤づくりの取り組み事例集
coconala_engineer
0
150
Jetpack Compose Modifier 徹底解説 / Jetpack Compose Modifier
wiroha
0
200
エンジニア視点で見る、 組織で運用されるデザインシステムにするには
shunya078
1
310
Featured
See All Featured
Side Projects
sachag
451
42k
Making Projects Easy
brettharned
113
5.8k
What's new in Ruby 2.0
geeforr
340
31k
No one is an island. Learnings from fostering a developers community.
thoeni
18
2.9k
Rebuilding a faster, lazier Slack
samanthasiow
78
8.6k
Debugging Ruby Performance
tmm1
72
12k
Fashionably flexible responsive web design (full day workshop)
malarkey
401
65k
How STYLIGHT went responsive
nonsquared
93
5.1k
Build The Right Thing And Hit Your Dates
maggiecrowley
30
2.3k
Teambox: Starting and Learning
jrom
131
8.7k
The Psychology of Web Performance [Beyond Tellerrand 2023]
tammyeverts
36
2.1k
Embracing the Ebb and Flow
colly
83
4.4k
Transcript
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything