Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Sign up for free
Menu
Search
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Features
All features
Private URLs
Password Protection
Custom URLS
Scheduled publishing
Remove Branding
Restrict embedding
Deck Collections
Notes
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Explore
Featured decks
Featured speakers
Programming
Technology
Storyboards
Pricing
Search
Sign in
Sign up for free
Event-Driven Applications: Apache Kafka and Python
Search
FTisiot
March 24, 2021
Technology
1k
0
Share
Embed
Copy iframe code
Copy JS code
Copy link
Start on current slide
Event-Driven Applications: Apache Kafka and Python
How to create Event-Driven Applications with Apache Kafka and Python
FTisiot
March 24, 2021
Transcript
30 slides
Event-Driven Applications: Apache Kafka and Python Francesco Tisiot - Developer
Advocate @ftisiot
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Why Kafka?
@ftisiot Event Driven Applications
@ftisiot What is an Event?
@ftisiot Why Event-driven?
@ftisiot What is Kafka? Topic A Topic B 0 1
2 3 4 0 1 2 3
@ftisiot Distributed Brokers Replication Factor 3 2
@ftisiot Distributed Replication Factor 2
@ftisiot What is a Kafka Event? <K, V> <"temp_max", 35.3>
<{ shop: "Mario's Pizza", phone_line: 3 } { orderId: 123456 name: "Andrea Pirlo", pizzas: [ { pizza: "Margherita", additionalToppings: ["Tuna","Onions","Ham"] }, ... ] } , >
@ftisiot Write to Kafka Producer Topic A Hostname + Port
Authentication Encoding
@ftisiot Hostname + Port Authentication Decoding Topic Name(s) Read from
Kafka Consumer Offset 0 1 2 3
@ftisiot Pizza Demo!
@ftisiot Log Size
@ftisiot Topic Partitions
@ftisiot Distributed
@ftisiot Topic Retention Policies Time Size
@ftisiot Selecting a Partition <K, V> Same Key Same Partition
@ftisiot Ordering 2 1 3 2 1 3 Partition 0
Partition 1 Partition 1 Guaranteed per Partition
@ftisiot Scaling Out P1 P2 P3 C1 C2
@ftisiot Partitions Demo!
@ftisiot Multiple Applications C1 C2 Consumer Group 1 CX Consumer
Group 2
@ftisiot Consumer Groups Demo!
@ftisiot Kafka Connect Kafka Connect Kafka Connect
@ftisiot Evolve Existing Applications Kafka Connect CDC
@ftisiot Distribute Events Config File Only!
@ftisiot Kafka Connect Demo!
@ftisiot Resources https://github.com/aiven/kafka-python-fake-data-producer https://aiven.io/ @ftisiot https://github.com/aiven/kafka-python-notebooks
@ftisiot Ask me Anything
More Decks by FTisiot
See All by FTisiot
Apache Kafka JDBC Source Connector: What could go wrong?
ftisiot
0
160
Solve the Knapsack problem with recursive queries and PostgreSQL
ftisiot
0
190
Build an Open Source Data Pipeline
ftisiot
0
370
Get to know Apache Kafka with Jupyter Notebooks
ftisiot
0
800
Fix Your Strings in PostgreSQL
ftisiot
0
310
Practical Tips & Tricks for Apache Kafka Messages Integration
ftisiot
0
280
Track Database Changes with Apache Kafka
ftisiot
0
330
Apache Kafka and Flink: Stateful Streaming Data Pipelines made easy with SQL
ftisiot
0
1k
Kickstart your Kafka with Faker Data
ftisiot
0
500
Other Decks in Technology
See All in Technology
マネジメントされる側の仕事は「文句」を言うこと
riyaamemiya
7
5.5k
ぼくのClaude Codeの使い方
falken
PRO
5
7.4k
「任されビリティ」で引き受けて「よしなりてぃ」に頼らず任せる / Accept tasks with "makasarebility" and hand them over without relying on "yoshinarily"
pauli
1
720
Sonnetだけで業務はまわる?
kintotechdev
1
140
IP重複は"ランディングVPC"で受け止める ― PrivateLink と Transit Gatewayによるマルチテナント接続基盤の設計
hatahata021
2
200
人間+AIを「チーム」としてプロセスを再設計する
littlehands
4
3.3k
プロダクションコードにおける encoding/json から encoding/json/v2 へのスムーズな移行
sansantech
PRO
1
440
3文字略語に騙されない! 世界一わかりやすい『EDR』超入門
kamadamakoto
1
200
これからのデータエンジニアの役割について考える / Rethinking the Role of Data Engineers in the AI Era
okiyuki99
0
520
パスワードのない鍵で、 データを手元に置く
ginjihayashi
0
220
OpenTelemetryで変わった 新時代のCloudWatchメトリクス監視
tsukuboshi
0
370
速くなった開発の効果をどう推し量るのか
bulbulpaul
0
120
Featured
See All Featured
Designing Powerful Visuals for Engaging Learning
tmiket
1
660
Building Better People: How to give real-time feedback that sticks.
wjessup
371
20k
Leading Effective Engineering Teams in the AI Era
addyosmani
9
2.9k
The Mindset for Success: Future Career Progression
greggifford
PRO
0
560
Google's AI Overviews - The New Search
badams
0
1.7k
Sam Torres - BigQuery for SEOs
techseoconnect
PRO
0
610
Creating an realtime collaboration tool: Agile Flush - .NET Oxford
marcduiker
35
2.6k
Why Mistakes Are the Best Teachers: Turning Failure into a Pathway for Growth
auna
0
350
<Decoding/> the Language of Devs - We Love SEO 2024
nikkihalliwell
1
380
Navigating Weather and Climate Data
rabernat
0
590
Effective software design: The role of men in debugging patriarchy in IT @ Voxxed Days AMS
baasie
1
610
We Have a Design System, Now What?
morganepeng
55
8.4k