Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Queues and the beanstalkd
Search
Daniel Cousineau
February 06, 2015
Programming
1
660
Queues and the beanstalkd
Given at Sunshine PHP 2015
Daniel Cousineau
February 06, 2015
Tweet
Share
More Decks by Daniel Cousineau
See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
560
React @ Scale
dcousineau
0
170
Frontend Performance & You
dcousineau
0
290
Feature Flags & You
dcousineau
2
96
React+Redux @ Scale
dcousineau
1
330
Reframing The Problem - DCJS July 2016
dcousineau
0
130
YAFT
dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
dcousineau
0
380
JavaScript for PHP Developers
dcousineau
4
700
Other Decks in Programming
See All in Programming
WebエンジニアがSwiftをブラウザで動かすプレイグラウンドを作ってみた
ohmori_yusuke
0
160
育てるアーキテクチャ:戦い抜くPythonマイクロサービスの設計と進化戦略
fujidomoe
1
140
Model Pollution
hschwentner
1
180
Web Components で実現する Hotwire とフロントエンドフレームワークの橋渡し / Bridging with Web Components
da1chi
2
1.2k
クラシルを支える技術と組織
rakutek
0
190
Pull-Requestの内容を1クリックで動作確認可能にするワークフロー
natmark
1
310
ネイティブ製ガントチャートUIを作って学ぶUICollectionViewLayoutの威力
jrsaruo
0
110
そのpreloadは必要?見過ごされたpreloadが技術的負債として爆発した日
mugitti9
2
2.5k
実践AIチャットボットUI実装入門
syumai
7
2.3k
猫と暮らすネットワークカメラ生活🐈 ~Vision frameworkでペットを愛でよう~ / iOSDC Japan 2025
yutailang0119
0
210
Server Less Code More - コードを書かない時代に生きるサーバーレスデザイン / server-less-code-more
gawa
5
1.9k
Swiftビルド弾丸ツアー - Swift Buildが作る新しいエコシステム
giginet
PRO
0
1.5k
Featured
See All Featured
Chrome DevTools: State of the Union 2024 - Debugging React & Beyond
addyosmani
7
880
The Invisible Side of Design
smashingmag
301
51k
How to Think Like a Performance Engineer
csswizardry
27
2k
Documentation Writing (for coders)
carmenintech
75
5k
4 Signs Your Business is Dying
shpigford
185
22k
It's Worth the Effort
3n
187
28k
The Web Performance Landscape in 2024 [PerfNow 2024]
tammyeverts
9
840
Balancing Empowerment & Direction
lara
4
660
Let's Do A Bunch of Simple Stuff to Make Websites Faster
chriscoyier
507
140k
The Success of Rails: Ensuring Growth for the Next 100 Years
eileencodes
46
7.6k
Scaling GitHub
holman
463
140k
Building Flexible Design Systems
yeseniaperezcruz
329
39k
Transcript
QUEUES & THE GIANT BEANSTALKD
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
@croscon
None
None
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE
CASE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:
The Data Structure
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous
Computation N seconds m Seconds n+m seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z
15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation N seconds m Seconds N seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Z 5 5 Fetch Z A Work
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 worker 1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
3 worker 1 worker 2 job 3
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker worker worker web app cron daemon db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
input output
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do
work update database or save file[s] or new job all data necessary to complete job
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize
image make thumbs Original Location thumbnail[s] image file new Name[s]
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute
stats account Report account id account data
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add
to master report account report
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post
tweet api response twitter id access tokens tweet text metadata
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update
database api response
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either
Read, or Write, never both
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If
you write, design for OVERWRITE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 B: 25 A: 10 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 A: 10 B: 25 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker web app cron db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler
Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its
fine to run a process let it die then resurrect it
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:
have long running process listen for jobs fork single-run job process
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep
jobs & workers small keep surface area for failures small
THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :
@DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15