Upgrade to Pro
— share decks privately, control downloads, hide ads and more …
Speaker Deck
Features
Speaker Deck
PRO
Sign in
Sign up for free
Search
Search
Queues and the beanstalkd
Search
Daniel Cousineau
February 06, 2015
Programming
1
650
Queues and the beanstalkd
Given at Sunshine PHP 2015
Daniel Cousineau
February 06, 2015
Tweet
Share
More Decks by Daniel Cousineau
See All by Daniel Cousineau
Time is a Social Construct
dcousineau
1
540
React @ Scale
dcousineau
0
160
Frontend Performance & You
dcousineau
0
270
Feature Flags & You
dcousineau
2
93
React+Redux @ Scale
dcousineau
1
320
Reframing The Problem - DCJS July 2016
dcousineau
0
120
YAFT
dcousineau
2
150
How Not Writing PHP Makes You Better At PHP
dcousineau
0
370
JavaScript for PHP Developers
dcousineau
4
690
Other Decks in Programming
See All in Programming
明示と暗黙 ー PHPとGoの インターフェイスの違いを知る
shimabox
2
170
C++20 射影変換
faithandbrave
0
500
第9回 情シス転職ミートアップ 株式会社IVRy(アイブリー)の紹介
ivry_presentationmaterials
1
200
生成AIコーディングとの向き合い方、AIと共創するという考え方 / How to deal with generative AI coding and the concept of co-creating with AI
seike460
PRO
1
320
たった 1 枚の PHP ファイルで実装する MCP サーバ / MCP Server with Vanilla PHP
okashoi
1
140
PHPで始める振る舞い駆動開発(Behaviour-Driven Development)
ohmori_yusuke
2
150
Railsアプリケーションと パフォーマンスチューニング ー 秒間5万リクエストの モバイルオーダーシステムを支える事例 ー Rubyセミナー 大阪
falcon8823
3
790
ASP.NETアプリケーションのモダナイズ インフラ編
tomokusaba
1
390
FormFlow - Build Stunning Multistep Forms
yceruto
1
190
関数型まつり2025登壇資料「関数プログラミングと再帰」
taisontsukada
2
840
Cursor AI Agentと伴走する アプリケーションの高速リプレイス
daisuketakeda
1
120
アンドパッドの Go 勉強会「 gopher 会」とその内容の紹介
andpad
0
250
Featured
See All Featured
Six Lessons from altMBA
skipperchong
28
3.8k
How to Think Like a Performance Engineer
csswizardry
24
1.7k
Fireside Chat
paigeccino
37
3.5k
Practical Orchestrator
shlominoach
188
11k
StorybookのUI Testing Handbookを読んだ
zakiyama
30
5.8k
Large-scale JavaScript Application Architecture
addyosmani
512
110k
Code Reviewing Like a Champion
maltzj
524
40k
Distributed Sagas: A Protocol for Coordinating Microservices
caitiem20
331
22k
XXLCSS - How to scale CSS and keep your sanity
sugarenia
248
1.3M
How To Stay Up To Date on Web Technology
chriscoyier
790
250k
[RailsConf 2023 Opening Keynote] The Magic of Rails
eileencodes
29
9.5k
Building an army of robots
kneath
306
45k
Transcript
QUEUES & THE GIANT BEANSTALKD
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM @dcousineau
@croscon
None
None
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM USE
CASE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM THEORY
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUE:
The Data Structure
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM FIFO
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Synchronous
Computation N seconds m Seconds n+m seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A
Fetch Z Work Z 5 10 10 Work 5 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Z
15 5 10 A Fetch Z Work 5 10 Work 15 Synchronous Computation
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation N seconds m Seconds N seconds
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Asynchronous
Computation Multi-Threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Race
Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Z 5 5 Fetch Z A Work
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Locking
A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locking A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Dead
Locks A B Fetch Z Z 5 5 Fetch Y A Y 10 B 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM QUEUING
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 worker 1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM job
3 worker 1 worker 2 job 3
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker worker worker web app cron daemon db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
input output
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM do
work update database or save file[s] or new job all data necessary to complete job
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM resize
image make thumbs Original Location thumbnail[s] image file new Name[s]
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM compute
stats account Report account id account data
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM add
to master report account report
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM post
tweet api response twitter id access tokens tweet text metadata
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM update
database api response
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM EXAMPLE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM http://kr.github.io/beanstalkd/
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://github.com/dcousineau/queues-example
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM TIPS
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Either
Read, or Write, never both
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM If
you write, design for OVERWRITE
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 B: 25 A: 10 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 worker 1 worker 2 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM A:
10 B: 20 B: 25 worker 1 worker 2 reserve A: 10 B: 20 A: ? B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM B:
25 worker 1 worker 2 A: 10 B: 25 B: 20
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 20 B: 25 Completed Reserve B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM worker
1 worker 2 A: 10 B: 25
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM queue
worker web app cron db
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM scheduler
Every X minutes, check: “What should be done NOW” worker db web app Assume DB is up to date
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM its
fine to run a process let it die then resurrect it
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM OR:
have long running process listen for jobs fork single-run job process
DANIEL COUSINEAU // FOLLOW ME : @DCOUSINEAU OR HTTP://DCOUSINEAU.COM Keep
jobs & workers small keep surface area for failures small
THANKS. FOR YOUR ATTENTION DANIEL COUSINEAU // FOLLOW ME :
@DCOUSINEAU OR HTTP://DCOUSINEAU.COM https://joind.in/13443 http://bit.ly/queues-ssp15