Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Scale like a pro
Search
Python Porto
December 14, 2017
Programming
1
270
Scale like a pro
Distributed task processing with Python and Celery
Python Porto
December 14, 2017
Tweet
Share
More Decks by Python Porto
See All by Python Porto
Detecting phishing with Recurrent Neural Networks
pyporto
0
42
Quick and Robust API with Django Rest Framework
pyporto
1
340
Django as your data management framework
pyporto
1
1.1k
Can my computer make jokes
pyporto
0
110
Building a serverless cloud service
pyporto
0
58
Python Porto #10. Past, present and future
pyporto
0
85
Entertaining testing with pytest
pyporto
0
190
Joyful Python Web App development with Appier
pyporto
0
160
Other Decks in Programming
See All in Programming
構造化・自動化・ガードレール - Vibe Coding実践記 -
tonegawa07
0
150
CLI ツールを Go ライブラリ として再実装する理由 / Why reimplement a CLI tool as a Go library
ktr_0731
3
570
それ CLI フレームワークがなくてもできるよ / Building CLI Tools Without Frameworks
orgachem
PRO
11
2.7k
中級グラフィックス入門~効率的なメッシュレット描画~
projectasura
3
1.7k
TypeScriptでDXを上げろ! Hono編
yusukebe
3
870
テストから始めるAgentic Coding 〜Claude Codeと共に行うTDD〜 / Agentic Coding starts with testing
rkaga
17
6.1k
Google I/O Extended Incheon 2025 ~ What's new in Android development tools
pluu
1
190
Reactの歴史を振り返る
tutinoko
1
140
Strands Agents で実現する名刺解析アーキテクチャ
omiya0555
1
110
The Evolution of Enterprise Java with Jakarta EE 11 and Beyond
ivargrimstad
0
520
レトロゲームから学ぶ通信技術の歴史
kimkim0106
0
130
AI コーディングエージェントの時代へ:JetBrains が描く開発の未来
masaruhr
2
220
Featured
See All Featured
Measuring & Analyzing Core Web Vitals
bluesmoon
7
530
Fashionably flexible responsive web design (full day workshop)
malarkey
407
66k
10 Git Anti Patterns You Should be Aware of
lemiorhan
PRO
656
60k
Dealing with People You Can't Stand - Big Design 2015
cassininazir
367
26k
Embracing the Ebb and Flow
colly
86
4.8k
Documentation Writing (for coders)
carmenintech
72
4.9k
Reflections from 52 weeks, 52 projects
jeffersonlam
351
21k
Gamification - CAS2011
davidbonilla
81
5.4k
How GitHub (no longer) Works
holman
314
140k
Site-Speed That Sticks
csswizardry
10
720
Understanding Cognitive Biases in Performance Measurement
bluesmoon
29
1.8k
Keith and Marios Guide to Fast Websites
keithpitt
411
22k
Transcript
Scale like a pro Distributed computing with message queues and
Python Roman Imankulov | Python Porto | December 2017
Source: https://blog.kissmetrics.com/wp-content/uploads/2011/04/loading-time.pdf
browser ./webserver.py request POST /obj/<id> obj.update()
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes send
email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” …
browser ./webserver.py request POST /obj/<id> obj.update() update search indexes update
business analytics send email “object updated” … response
None
browser ./webserver.py request POST /obj/<id> response obj.update() ./worker.py ./worker.py ./worker.py
update search indexes send email “object updated” update business analytics
Message queues
None
None
queue
frontends queue
frontends queue workers
frontends queue job 1 workers
frontends queue job 1 workers job 2
frontends queue job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue put() job 1 workers job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers put() job 2 job 3
frontends queue job 1 workers job 2 job 3
frontends queue job 1 get() workers job 2 job 3
frontends queue get() workers job 2 job 3 job 1
frontends queue workers job 2 job 3 job 1
frontends queue workers get() job 2 job 3 job 1
frontends queue workers get() job 3 job 1 job 2
frontends queue workers job 3 job 1 job 2
frontends queue workers job 3 job 2
frontends queue workers job 3 job 2 get()
frontends queue workers job 2 job 3 get()
frontends queue workers job 2 job 3
frontends queue workers
Queue in python
Multiprocessing
None
None
None
None
None
None
None
Celery The queue out of the box
None
None
None
None
None
Celery Workflows chains, groups and chords
Task signatures
Task signatures
Task signatures
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
Chains a(…) b(…) c(…)
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
a(…) b(…) c(…) Groups
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Chords a(…) b(…) c(…) d(…)
Celery Extras out of the box
None
• Different backends
• Different backends • Different serializers
• Different backends • Different serializers • Callbacks / errbacks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute)
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics
• Different backends • Different serializers • Callbacks / errbacks
• Progress reports from tasks • Delayed tasks • Ignored results • Expiring results • Retry policies • Time limits on task execution • Rate limits (N tasks per minute) • Autoscaling • Multiple queues • Introspection and statistics • Periodic tasks and crontabs
None
facebook.com/pyporto