Queues And The Giant beanstalkd

In the modern world of scaling web applications, queues have become a go-to for increasing performance by parallelizing and backgrounding heavy duty tasks. Learn practical usages of queues through beanstalkd.

Given at Feb 2013 DallasPHP meeting and ConFoo 2013

Transcript

1. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Queues and the Giant beanstalkd
2. Daniel Cousineau Developer at SplashMedia @dcousineau
3. use Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
4. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
5. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
6. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
7. Theory Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
8. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com QUEUE: The Data Structure
9. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com FIFO
10. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com synchronous Computation N seconds m Seconds n+m seconds
11. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous Computation A Fetch Z Work Z 5 5
12. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous Computation A Fetch Z Work Z 5 10 10 Work 5
13. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Synchronous Computation Z 15 5 10 A Fetch Z Work 5 10 Work 15
14. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Asynchronous Computation N seconds m Seconds N seconds
15. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Asynchronous Computation Multi-threading Multi-Process Memory Thread Thread Thread process process process memory Memory Memory
16. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race Condition A B Fetch Z Work Z 5 5 Work Fetch Z 5
17. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race Condition A B Fetch Z Work Z 5 10 Work 10 Work Fetch Z 5 5
18. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Race Condition A B Fetch Z Fetch Z Work Z 5 5 Work 10 Work 15 15 5 10
19. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking A B Fetch Z Z 5 5 Fetch Z A Work
20. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking A B Fetch Z Fetch Z Work Z 5 Work 15 15 15 B 5 A wait wait
21. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Locking A B Fetch Z Fetch Z Work Z 5 Work 15 15 Work 25 25 15 B 5 A wait wait
22. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD LOCKING A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10
23. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD LOCKS A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10 Fetch Y Fetch Z
24. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com DEAD LOCKS A B Fetch Z Z 5 5 Fetch Y A Y 10 b 10 Fetch Y Fetch Z wait wait wait wait wait wait wait wait wait
25. Execution Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
26. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job 1 job 2 worker 1 worker 2
27. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job 1 job 2 job 3 worker 1 worker 2 reserve job 1
28. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job 1 job 2 job 3 worker 1 worker 2 reserve job 1 job 2
29. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job 2 job 3 worker 1 worker 2 Completed Reserve job 3 job 2
30. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com job 3 worker 1 worker 2 job 3
31. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker 1 worker 2
32. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com queue worker worker worker web app CRON daemon db
33. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Worker input output
34. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com do work update database or save file[s] or new job all data necessary to complete job
35. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com resize image make thumbs Original Location thumbnail[s] image file new Name[s]
36. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com compute stats account Report account id account data
37. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com add to master report account report
38. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com post tweet api response twitter id access tokens tweet text metadata
39. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com update database api response
40. example Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
41. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com http://kr.github.com/beanstalkd
42. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
43. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
44. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
45. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com https://github.com/dcousineau/queues-example
46. Special Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
47. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com Either Read, or Write, never both
48. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com If you write, design for overwrite
49. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 worker 1 worker 2 A: ? B: ?
50. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 A: ? B: ?
51. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 B: 20 A: ? B: ?
52. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com B: 20 B: 25 worker 1 worker 2 Completed Reserve B: 25 B: 20 A: 10 B: ?
53. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com B: 25 worker 1 worker 2 B: 25 A: 10 B: 20
54. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker 1 worker 2 A: 10 B: 25
55. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 worker 1 worker 2 A: ? B: ?
56. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 A: ? B: ?
57. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com A: 10 B: 20 B: 25 worker 1 worker 2 reserve a: 10 B: 20 A: ? B: ?
58. Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com worker 1 worker 2 B: 25 B: 20 A: 10 B: ? A: 10 B: 20 B: 25 STUCK!
59. Q& A Put your questions Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com
60. THANKS. FOR YOUR ATTENTION Daniel Cousineau // follow me : @dcousineau or http://dcousineau.com http://joind.in/7873