Scale-Independent Python

Have you ever written an application only to find yourself rewriting it later so it can scale? Many of us have. In this talk, you’ll learn scale-independent computing; what it is and how it can help you more easily scale your applications.

Scaling applications is hard. Our typical approaches all come with tradeoffs that we wish we didn’t have to make. There is a better way, scale-independent computing. Based on ideas from Pat Helland’s paper “Life Beyond Distributed Transactions,” scale-independent computing allows application developers to write their code without regard to scale and then run it on a scale-aware platform. The idea is relatively simple, the code you write should be able to run across any number of Python processes without you having to change any of your code. During this talk, you’ll learn the core ideas of scale-independence and see an example scale-independent application in action.

Transcript

1. SCALE-INDEPENDENT PYTHON HOW TO SCALE YOUR PYTHON APPLICATION WITHOUT ANY CODE CHANGES
2. SEAN T. ALLEN VP OF ENGINEERING AT WALLAROO LABS AUTHOR OF “STORM APPLIED” @SEANTALLEN @WALLAROOLABS
3. THIS TALK IS ABOUT “SCALING”
4. “SCALING” IS ABOUT REACTING TO CHANGE
5. LET’S TALK ABOUT REACTING TO CHANGES IN DATA
6. LET’S TALK ABOUT CHANGES IN APPLICATION STATE
7. LET’S TALK ABOUT A VERY SPECIFIC THING
8. LET’S TALK ABOUT HANDLING “MORE THINGS”
9. MORE THINGS?
10. MORE CUSTOMERS
11. MORE ORDERS
12. MORE VENDORS
13. MORE THINGS
14. THAT’S THE “SCALING” WE’RE TALKING ABOUT
15. WE’RE TALKING ABOUT HORIZONTAL SCALING
16. WE’RE TALKING ABOUT PAT HELLAND’S LIFE BEYOND DISTRIBUTED TRANSACTIONS
17. SEVERAL BIG IDEAS
18. LET’S TALK ABOUT ONE BIG IDEA
19. TWO-LAYER ARCHITECTURE
20. SCALE-INDEPENDENT LAYER
21. OUR BUSINESS LOGIC LIVES IN THE SCALE-INDEPENDENT LAYER
22. SCALE-AWARE LAYER
23. CLUSTERING
24. MESSAGE ROUTING
25. GUARANTEED MESSAGE DELIVERY
26. FAULT TOLERANCE
27. AND AN API TO CONNECT YOUR SCALE-INDEPENDENT CODE TO THE SCALE-AWARE LAYER
28. AND THAT IS WHAT WE’VE BUILT AT WALLAROO LABS
29. WALLAROO SCALE-INDEPENDENT COMPUTING FOR PYTHON
30. STATEFUL DATA PROCESSING ENGINE
31. SCALE-INDEPENDENT API
32. FLOW OF DATA
33. TRANSFORMATIONS OF DATA
34. ACCUMULATED APPLICATION STATE
35. APPLICATION DEFINITION FLOW OF DATA FROM SOURCE TO SINK
36. TWO DATA PIPELINES ORDERS
37. TWO DATA PIPELINES MARKET DATA
38. DEFINE OUR SOURCES 1 PER PIPELINE
39. DEFINE OUR OPERATIONS IN THIS EXAMPLE, THERE’S 1 PER PIPELINE
40. DEFINE OUR OPERATIONS CHECK ORDER AGAINST SYMBOL DATA
41. DEFINE OUR OPERATIONS UPDATE SYMBOL DATA WITH LATEST MARKET DATA
42. DEFINE OUR SINKS 1 PER PIPELINE
43. DEFINE OUR SINKS ORDERS PIPELINE MIGHT HAVE OUTPUT
44. DEFINE OUR SINKS MARKET DATA ONLY UPDATES SYMBOL DATA- NO OUTPUT
45. SCALE-INDEPENDENT ONLY FLOW OF DATA AND OPERATIONS
46. 100% SCALE-INDEPENDENT NOTHING ABOUT THE NUMBER OF PYTHON PROCESSES
47. RUN WITH ANY NUMBER OF PYTHON PROCESSES
48. WITHOUT CHANGING CODE
49. SUCH IS THE GLORY OF SCALE-INDEPENDENCE
50. THERE’S MORE BUT WE ARE PROBABLY CLOSE TO BEING OUT OF TIME
51. QUICKLY, A FEW MORE WALLAROO FEATURES
52. PLAIN OLD PYTHON FUNCTIONS
53. PLAIN OLD PYTHON DATA OBJECTS
54. USE ANY PYTHON LIBRARY
55. LEARN MORE GITHUB.COM/SEANTALLEN/ SCALE-INDEPENDENT-PYTHON