Python and Asynchrony

Presented at Jobspace on 14 October, 2016.

An introduction to asynchrony programming in Python. This presentation consists of two parts: First I introduce the idea behind asynchrony, and how it differs from more “traditional” (in Python sense) concurrent solutions.

In the second part I use code examples to walk through the development of async paradigms, introducing async through callbacks, promises, and finally coroutine, so we get a deeper understanding how the underlying plumbing works when you run an async program.

Code examples for the second part of the presentation can be found at http://bit.ly/asyncio-examples.

Transcript

1. Python and Asynchrony
2. http://bit.ly/asyncio-examples
3. Quick Questions • Concurrency with Python • Threads and multi-processing • Single-thread asynchrony • asyncio, Twisted, Tornado, etc.
4. Me • Call me TP • Follow @uranusjr • https://uranusjr.com
7. http://macdown.uranusjr.com
8. www. .com
9. 10–11 June 2017 (Ծ) 7
11. Agenda • Introduction to asynchrony • Async the asyncio way • The road ahead
13. Waiting (wasted) Synchronous
14. Threads
15. Threads • More than you need • Race conditions • Dead-locking • Resource starvation
16. To share, or not to share: that is the question.
17. LET’S TALK ABOUT SOMETHING “DIFFERENT”
18. Context Switching • First as a CPU thing • Interrupts • Software interrupts
19. Multitasking • A common OS problem • More tasks than threads • Cooperative • Pre-emptive
21. Same problem?
22. Threads
23. Scheduling
24. Asynchronous
25. Sync vs. Async
26. ASYNCIO
27. Async How? • Callbacks • Promises • Coroutines
28. Event Loop
29. Launch Initialise Running Cleanup Terminate Register callbacks Receive events
30. import asyncio # Init code goes here. loop = asyncio.get_event_loop() loop.run_forever()
31. import asyncio # Init code goes here. def print_and_exit(): print('Hello asyncio!') asyncio.get_event_loop().stop() loop = asyncio.get_event_loop() loop.call_soon(print_and_exit) loop.run_forever()
32. CALLBACK When the going gets rough.
33. Pyramid of Doom $(function () { $("button").click(function (e) { $.get(URL, function (data) { $(".list").each(function () { $(this).click(function () { setTimeout(function () { alert("Top of the world"); }, 1000); }); }); }); }); });
34. WELCOME TO CALLBACK HELL
35. def send_repos_request(): ... task = ... task.add_done_callback(...) def read_repos(future): ... = future.result() ... task = ... task.add_done_callback(...) def send_forked_repo_requests(future): ... = future.result() ... task = ... task.add_done_callback(...) def read_forked_repos(future): ... = future.result() ... task = ... task.add_done_callback(...)
36. Promise
37. def main(): (send_repos_request() .then(read_repos) .then(send_forked_repo_requests, cleanup) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
38. def main(): (send_repos_request() .then(read_repos) .then(send_forked_repo_requests, cleanup) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
39. def main(): (send_repos_request() .then(read_repos) .then(send_forked_repo_requests, cleanup) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
40. def main(): (send_repos_request() .then(read_repos) .then(send_forked_repo_requests, cleanup) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
41. def main(): (send_repos_request() .then(read_repos) .catch(cleanup) .then(send_forked_repo_requests) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
42. def main(): (send_repos_request() .then(read_repos) .catch(cleanup) .then(send_forked_repo_requests) .then(read_forked_repo_requests) .then(print_source_names) .done(cleanup))
43. ???????
44. Coroutine
45. async def run(session): response = await send_repos_request(session) try: data = await read_repos(response) except APIError: return responses = await send_forked_repo_requests(session, data) datasets = await read_forked_repo_requests(responses) print_source_names(datasets)
46. async def run(session): response = await send_repos_request(session) try: data = await read_repos(response) except APIError: return responses = await send_forked_repo_requests(session, data) datasets = await read_forked_repo_requests(responses) print_source_names(datasets)
48. Everything Has a Counterpart • async def / await • async with • async for (Python 3.6)
49. class AsyncContextManager: async def __aenter__(self): await log('entering context') async def __aexit__(self, exc_type, exc, tb): await log('exiting context')
50. class AsyncIterable: def __aiter__(self): return self async def __anext__(self): data = await ... if data: return data else: raise StopAsyncIteration
51. Python asyncio.
52. Wrapping Up • What is asynchrony • How asynchrony works • Asynchrony and asyncio
53. Further Watching • A gentle introduction to asyncio • What in the World is Asyncio? • Topics of Interest (Python Asyncio) • You Might Not Want Async • and more…!