Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Generators: The Final Frontier

Generators: The Final Frontier

Tutorial. PyCon 2014. Montreal. Conference video at https://www.youtube.com/watch?v=D1twn9kLmYg. Screencast at https://www.youtube.com/watch?v=5-qadlG7tWo

David Beazley

April 10, 2014
Tweet

More Decks by David Beazley

Other Decks in Programming

Transcript

  1. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators: The Final

    Frontier David Beazley (@dabeaz) http://www.dabeaz.com Presented at PyCon'2014, Montreal 1
  2. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators

    2 • Generator Tricks for Systems Programmers (2008) http://www.dabeaz.com/generators/ • A flying leap into generator awesomeness
  3. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Previously on Generators

    3 • A Curious Course on Coroutines and Concurrency (2009) http://www.dabeaz.com/coroutines/ • Wait, wait? There's more than iteration?
  4. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Today's Installment 4

    • Everything else you ever wanted to know about generators, but were afraid to try • Part 3 of a trilogy
  5. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Requirements 5 •

    You need Python 3.4 or newer • No third party extensions • Code samples and notes http://www.dabeaz.com/finalgenerator/ • Follow along if you dare!
  6. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Disclaimer 6 •

    This is an advanced tutorial • Assumes general awareness of • Core Python language features • Iterators/generators • Decorators • Common programming patterns • I learned a LOT preparing this
  7. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Will I Be

    Lost? 7 • Although this is the third part of a series, it's mostly a stand-alone tutorial • If you've seen prior tutorials, that's great • If not, don't sweat it • Be aware that we're focused on a specific use of generators (you just won't get complete picture)
  8. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Focus 8 •

    Material in this tutorial is probably not immediately applicable to your day job • More thought provoking and mind expanding • from __future__ import future practical utility bleeding edge
  9. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part I 9

    Preliminaries - Generators and Coroutines (rock)
  10. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators 101 •

    yield statement defines a generator function 10 def countdown(n): while n > 0: yield n n -= 1 • You typically use it to feed iteration for x in countdown(10): print('T-minus', x) • A simple, yet elegant idea
  11. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Under the Covers

    11 • Generator object runs in response to next() >>> c = countdown(3) >>> c <generator object countdown at 0x10064f900> >>> next(c) 3 >>> next(c) 2 >>> next(c) 1 >>> next(c) Traceback (most recent call last): File "<stdin>", line 1, in ? StopIteration >>> • StopIteration raised when function returns
  12. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 12 •

    Generators as "iterators" misses the big picture • There is so much more to yield
  13. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generators as Pipelines

    • Stacked generators result in processing pipelines • Similar to shell pipes in Unix 13 generator input sequence for x in s: generator generator def process(sequence): for s in sequence: ... do something ... yield item • Incredibly useful (see prior tutorial)
  14. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines 101 •

    yield can receive a value instead 14 def receiver(): while True: item = yield print('Got', item) • It defines a generator that you send things to recv = receiver() next(recv) # Advance to first yield recv.send('Hello') recv.send('World')
  15. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Coroutines and Dataflow

    15 • Coroutines enable dataflow style processing source coroutine coroutine send() send() • Publish/subscribe, event simulation, etc. coroutine coroutine send() send() coroutine send()
  16. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Fundamentals • The

    yield statement defines a generator function 16 def generator(): ... ... yield ... ... • The mere presence of yield anywhere is enough • Calling the function creates a generator instance >>> g = generator() >>> g <generator object generator at 0x10064f120> >>>
  17. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advancing a Generator

    • next(gen) - Advances to the next yield 17 def generator(): ... ... yield item ... • Returns the yielded item (if any) • It's the only allowed operation on a newly created generator • Note: Same as gen.__next__()
  18. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Sending to a

    Generator • gen.send(item) - Send an item to a generator 18 def generator(): ... item = yield ... ... yield value • Wakes at last yield, returns sent value • Runs to the next yield and emits the value g = generator() next(g) # Advance to yield value = g.send(item)
  19. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Closing a Generator

    • gen.close() - Terminate a generator 19 def generator(): ... try: yield except GeneratorExit: # Shutting down ... • Raises GeneratorExit at the yield • Only allowed action is to return • If uncaught, generator silently terminates g = generator() next(g) # Advance to yield g.close() # Terminate
  20. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Raising Exceptions •

    gen.throw(typ [, val [,tb]]) - Throw exception 20 def generator(): ... try: yield except RuntimeError as e: ... ... yield val • Raises exception at yield • Returns the next yielded value (if any) g = generator() next(g) # Advance to yield val = g.throw(RuntimeError, 'Broken')
  21. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Return Values

    • StopIteration raised on generator exit 21 def generator(): ... yield ... return result • Return value (if any) passed with exception • Note: Python 3 only behavior (in Python 2, generators can't return values) g = generator() try: next(g) except StopIteration as e: result = e.value
  22. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Delegation •

    yield from gen - Delegate to a subgenerator 22 def generator(): ... yield value ... return result def func(): result = yield from generator() • Allows generators to call other generators • Operations take place at the current yield • Return value (if any) is returned
  23. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Delegation Example •

    Chain iterables together 23 def chain(x, y): yield from x yield from y • Example: >>> a = [1, 2, 3] >>> b = [4, 5, 6] >>> for x in chain(a, b): ... print(x,end=' ') ... 1 2 3 4 5 6 >>> c = [7,8,9] >>> for x in chain(a, chain(b, c)): ... print(x, end=' ') ... 1 2 3 4 5 6 7 8 9 >>>
  24. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Mini-Reference • Generator

    instance operations 24 gen = generator() next(gen) # Advance to next yield gen.send(item) # Send an item gen.close() # Terminate gen.throw(exc, val, tb) # Raise exception result = yield from gen # Delegate • Using these, you can do a lot of neat stuff • Generator definition def generator(): ... yield ... return result
  25. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Common Motif

    • Consider the following 26 f = open() ... f.close() lock.acquire() ... lock.release() db.start_transaction() ... db.commit() start = time.time() ... end = time.time() • It's so common, you'll see it everywhere!
  26. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Managers •

    The 'with' statement 27 with open(filename) as f: statement statement ... with lock: statement statement ... • Allows control over entry/exit of a code block • Typical use: everything on the previous slide
  27. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management •

    It's easy to make your own (@contextmanager) 28 import time from contextlib import contextmanager @contextmanager def timethis(label): start = time.time() try: yield finally: end = time.time() print('%s: %0.3f' % (label, end-start) • This times a block of statements
  28. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management 29

    • Usage with timethis('counting'): n = 1000000 while n > 0: n -= 1 • Output counting: 0.023
  29. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management •

    Another example: temporary directories 30 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ...
  30. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Whoa, Whoa, Stop!

    • Another example: temporary directories 31 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): outdir = tempfile.mkdtemp() try: yield outdir finally: shutil.rmtree(outdir) • Example with tempdir() as dirname: ... What is this? • Not iteration • Not dataflow • Not concurrency • ????
  31. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Management •

    Under the covers 32 with obj: statements statements statements ... statements • If an object implements these methods it can monitor entry/exit to the code block obj.__enter__() obj.__exit__()
  32. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager •

    Implementation template 33 class Manager(object): def __enter__(self): return value def __exit__(self, exc_type, val, tb): if exc_type is None: return else: # Handle an exception (if you want) return True if handled else False • Use: with Manager() as value: statements statements
  33. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Context Manager Example

    • Automatically deleted temp directories 34 import tempfile import shutil class tempdir(object): def __enter__(self): self.dirname = tempfile.mkdtemp() return self.dirname def __exit__(self, exc, val, tb): shutil.rmtree(self.dirname) • Use: with tempdir() as dirname: ...
  34. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Alternate Formulation •

    @contextmanager is just a reformulation 35 import tempfile, shutil from contextlib import contextmanager @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • It's the same code, glued together differently
  35. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • How

    does it work? 36 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname finally: shutil.rmtree(dirname) • Think of "yield" as scissors • Cuts the function in half
  36. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • Each

    half maps to context manager methods 37 @contextmanager def tempdir(): dirname = tempfile.mkdtemp() try: yield dirname statements statements statements ... finally: shutil.rmtree(dirname) __enter__ __exit__ user statements ('with' block) • yield is the magic that makes it possible
  37. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • There

    is a wrapper class (Context Manager) 38 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): ... def __exit__(self, exc, val, tb): ... • And a decorator def contextmanager(func): def run(*args, **kwargs): return GeneratorCM(func(*args, **kwargs)) return run
  38. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • enter

    - Run the generator to the yield 39 class GeneratorCM(object): def __init__(self, gen): self.gen = gen def __enter__(self): return next(self.gen) def __exit__(self, exc, val, tb): ... • It runs a single "iteration" step • Returns the yielded value (if any)
  39. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Deconstruction • exit

    - Resumes the generator 40 class GeneratorCM(object): ... def __exit__(self, etype, val, tb): try: if etype is None: next(self.gen) else: self.gen.throw(etype, val, tb) raise RuntimeError("Generator didn't stop") except StopIteration: return True except: if sys.exc_info()[1] is not val: raise • Either resumes it normally or raises exception
  40. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Full Disclosure •

    Actual implementation is more complicated • There are some nasty corner cases • Exceptions with no associated value • StopIteration raised inside a with-block • Exceptions raised in context manager • Read source and see PEP-343 41
  41. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Discussion • Why

    start with this example? • A completely different use of yield • Being used to reformulate control-flow • It simplifies programming for others (easy definition of context managers) • Maybe there's more... (of course there is) 42
  42. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Async Processing •

    Consider the following execution model 45 def func(args): ... ... ... return result run_async(func, args) main thread • Examples: Run in separate process or thread, time delay, in response to event, etc.
  43. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example: Thread Pool

    46 from concurrent.futures import ThreadPoolExecutor def func(x, y): 'Some function. Nothing too interesting' import time time.sleep(5) return x + y pool = ThreadPoolExecutor(max_workers=8) fut = pool.submit(func, 2, 3) r = fut.result() print('Got:', r) • Runs the function in a separate thread • Waits for a result
  44. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 47 >>>

    fut = pool.submit(func, 2, 3) >>> fut <Future at 0x1011e6cf8 state=running> >>> • Future - A result to be computed later • You can wait for the result to return >>> fut.result() 5 >>> • However, this blocks the caller
  45. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures 48 def

    run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): result = fut.result() print('Got:', result) • Alternatively, you can register a callback • Triggered upon completion
  46. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exceptions 49 >>>

    fut = pool.submit(func, 2, 'Hello') >>> fut.result() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 395, in result return self.__get_result() File "/usr/local/lib/python3.4/concurrent/futures/_base.py", line 354, in __get_result raise self._exception File "/usr/local/lib/python3.4/concurrent/futures/thread.py", line 54, in run result = self.fn(*self.args, **self.kwargs) File "future2.py", line 6, in func return x + y TypeError: unsupported operand type(s) for +: 'int' and 'str' >>>
  47. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Futures w/Errors 50

    def run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Error handling with callbacks • Exception propagates out of fut.result() method
  48. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 51 def

    run(): fut = pool.submit(func, 2, 3) fut.add_done_callback(result_handler) def result_handler(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • Consider the structure of code using futures • Meditate on it... focus on the code. • This seems sort of familiar
  49. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Interlude 53 def

    entry(): fut = pool.submit(func, 2, 3) fut.add_done_callback(exit) def exit(fut): try: result = fut.result() print('Got:', result) except Exception as e: print('Failed: %s: %s' % (type(e).__name__, e)) • What if the function names are changed? • Wait! This is almost a context manager (yes)
  50. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Futures 54

    @inlined_future def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) run_inline_future(do_func) • Thought: Maybe you could do that yield trick • The extra callback function is eliminated • Now, just one "simple" function • Inspired by @contextmanager
  51. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Preview 57 t

    = Task(gen) • There are two separate parts • Part 1: Wrapping generators with a "task" • Part 2: Implementing some runtime code run_inline_future(gen) • Forewarning: It will bend your mind a bit
  52. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 58 •

    Will continue to use threads for examples • Mainly because they're easy to work with • And I don't want to get sucked into an event loop • Don't dwell on it too much • Key thing: There is some background processing
  53. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    59 def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) • Problem: Stepping through a generator • Involves gluing callbacks and yields together def do_func(x, y): yield pool.submit(func, x, y) result = print('Got:', result) add_done_callback() cut enter exit
  54. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    60 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result)
  55. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    61 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Task class wraps around and represents a running generator.
  56. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    62 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Advance the generator to the next yield, sending in a value (if any)
  57. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    63 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Attach a callback to the produced Future
  58. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running the Generator

    64 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None): try: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): result = fut.result() ! self.step(result) Collect result and send back into the generator
  59. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work?

    65 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_func(x, y): result = yield pool.submit(func, x, y) print('Got:', result) t = Task(do_func(2, 3)) t.step() • Try it: • Output: Got: 5 • Yes, it works Note: must initiate first step of the task to get it to run
  60. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Does it Work?

    66 pool = ThreadPoolExecutor(max_workers=8) def func(x, y): time.sleep(1) return x + y def do_many(n): while n > 0: result = yield pool.submit(func, n, n) print('Got:', result) n -= 1 t = Task(do_many(10)) t.step() • More advanced: multiple yields/looping • Yes, this works too.
  61. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 67

    class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc)
  62. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 68

    class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) send() or throw() depending on success
  63. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Exception Handling 69

    class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass def _wakeup(self, fut): try: result = fut.result() self.step(result, None) except Exception as exc: self.step(None, exc) Catch exceptions and pass to next step as appropriate
  64. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Error Example 70

    def do_func(x, y): try: result = yield pool.submit(func, x, y) print('Got:', result) except Exception as e: print('Failed:', repr(e)) t = Task(do_func(2, 'Hello')) t.step() • Try it: • Output: Failed: TypeError("unsupported operand type(s) for +: 'int' and 'str'",) • Yep, that works too.
  65. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Commentary 71 •

    This whole thing is rather bizarre • Execution of the inlined future takes place all on its own (concurrently with other code) • The normal rules don't apply
  66. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Consider 72 def

    recursive(n): yield pool.submit(time.sleep, 0.001) print('Tick:', n) Task(recursive(n+1)).step() Task(recursive(0)).step() • Infinite recursion? • Output: Tick: 0 Tick: 1 Tick: 2 ... Tick: 1662773 Tick: 1662774 ...
  67. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Part 4 73

    yield from yield from yield from yield from future (maybe) source: @UrsulaWJ
  68. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus

    74 • Focus on the future • Not the past • Not now • Yes, the future. • No, really, the future. (but not the singularity)
  69. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Singular Focus

    75 class Task: def __init__(self, gen): self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: pass ... generator must only produce Futures
  70. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 76 def

    after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • It's trying to delay the execution of a user- supplied inlined future until later.
  71. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 77 def

    after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • No Traceback (most recent call last): ... AttributeError: 'generator' object has no attribute 'add_done_callback'
  72. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler 78 def

    after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) yield gen Task(after(10, do_func(2, 3))).step() • Can you make library functions? • This is busted • gen is a generator, not a Future
  73. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt)

    79 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • Idea: Just iterate the generator manually • Make it produce the required Futures
  74. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (2nd Attempt)

    80 def after(delay, gen): ''' Run an inlined future after a time delay ''' yield pool.submit(time.sleep, delay) for f in gen: yield f Task(after(10, do_func(2, 3))).step() • What about this? • No luck. The result gets lost somewhere Got: None • Hmmm.
  75. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt)

    81 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5
  76. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (3rd Attempt)

    82 def after(delay, gen): yield pool.submit(time.sleep, delay) result = None try: while True: f = gen.send(result) result = yield f except StopIteration: pass Task(after(10, do_func(2, 3))).step() • Obvious solution (duh!) • Hey, it works! Got: 5 manual running of generator with results (ugh!)
  77. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (4th Attempt)

    83 def after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • A better solution: yield from • 'yield from' - Runs the generator for you Got: 5 • And it works! (yay!) • Awesome
  78. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com PEP 380 •

    yield from gen - Delegate to a subgenerator 84 def generator(): ... yield value ... return result def func(): result = yield from generator() • Transfer control to other generators • Operations take place at the current yield • Far more powerful than you might think
  79. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Conundrum 85 def

    after(delay, gen): yield pool.submit(time.sleep, delay) yield from gen • "yield" and "yield from"? • Two different yields in the same function • Nobody will find that confusing (NOT!)
  80. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt)

    86 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always!
  81. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Puzzler (5th Attempt)

    87 def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen Task(after(10, do_func(2, 3))).step() • Maybe this will work? • Just use 'yield from'- always! • No. 'yield' and 'yield from' not interchangeable: Traceback (most recent call last): ... TypeError: 'Future' object is not iterable >>>
  82. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Iterable Futures 89

    def patch_future(cls): def __iter__(self): ! if not self.done(): yield self return self.result() cls.__iter__ = __iter__ from concurrent.futures import Future patch_future(Future) • A simple ingenious patch • It makes all Future instances iterable • They simply produce themselves and the result • It magically makes 'yield from' work!
  83. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com All Roads Lead

    to Future 90 yield self • Future is the only thing that actually yields • Everything else delegates using 'yield from' • Future terminates the chain generator generator generator Future yield from yield from yield from run() next(gen)
  84. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Decorator 91

    import inspect def inlined_future(func): assert inspect.isgeneratorfunction(func) return func • Generators yielding futures is its own world • Probably a good idea to have some demarcation • Does nothing much at all, but serves as syntax @inlined_future def after(delay, gen): yield from pool.submit(time.sleep, delay) yield from gen • Alerts others about what you're doing
  85. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Wrangling 92

    t = Task(gen) t.step() • The "Task" object is just weird • No way to obtain a result • No way to join with it • Or do much of anything useful at all Task runs ???? t.step()
  86. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures

    93 class Task(Future): def __init__(self, gen): ! super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting
  87. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tasks as Futures

    94 class Task(Future): def __init__(self, gen): ! super().__init__() ! self._gen = gen def step(self, value=None, exc=None): try: if exc: fut = self._gen.throw(exc) else: fut = self._gen.send(value) fut.add_done_callback(self._wakeup) except StopIteration as exc: self.set_result(exc.value) • This tiny tweak makes it much more interesting A Task is a Future Set its result upon completion
  88. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 95 @inlined_future

    def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) • Obtaining the result of task • So, you create a task that runs a generator producing Futures • The task is also a Future • Right. Got it.
  89. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 96 @inlined_future

    def do_func(x, y): result = yield pool.submit(func, x, y) return result t = Task(do_func(2,3)) t.step() ... print("Got:", t.result()) class Task(Future): ... def step(self, value=None, exc=None): try: ... except StopIteration as exc: self.set_result(exc.value) ...
  90. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Task Runners 97

    def start_inline_future(fut): t = Task(fut) t.step() return t def run_inline_future(fut): t = start_inline_future(fut) return t.result() • You can make utility functions to hide details result = run_inline_future(do_func(2,3)) print('Got:', result) • Example: Run an inline future to completion • Example: Run inline futures in parallel t1 = start_inline_future(do_func(2, 3)) t2 = start_inline_future(do_func(4, 5)) result1 = t1.result() result2 = t2.result()
  91. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Step Back Slowly

    98 • Built a generator-based task system for threads Tasks @inline_future run_inline_future() Threads pools concurrent.future Futures submit result • Execution of the future hidden in background • Note: that was on purpose (for now)
  92. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 99 •

    Ideas are the foundation asyncio coroutines Tasks @coroutine run_until_complete() Event Loop asyncio Futures result • In fact, it's almost exactly the same • Naturally, there are some details with event loop
  93. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Simple Example 100

    import asyncio def func(x, y): return x + y @asyncio.coroutine def do_func(x, y): yield from asyncio.sleep(1) return func(x, y) loop = asyncio.get_event_loop() result = loop.run_until_complete(do_func(2,3)) print("Got:", result) • asyncio "hello world"
  94. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 101

    import asyncio @asyncio.coroutine def echo_client(reader, writer): while True: line = yield from reader.readline() if not line: break resp = b'Got:' + line writer.write(resp) writer.close() loop = asyncio.get_event_loop() loop.run_until_complete( asyncio.start_server(echo_client, host='', port=25000) ) loop.run_forever() • asyncio - Echo Server
  95. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Python Threads 105

    • Threads, what are they good for? • Answer: Nothing, that's what! • Damn you GIL!!
  96. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actually... 106 •

    Threads are great at doing nothing! time.sleep(2) # Do nothing for awhile data = sock.recv(nbytes) # Wait around for data data = f.read(nbytes) • In fact, great for I/O! • Mostly just sitting around
  97. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 107

    • Threads are weak for computation • Global interpreter lock only allows 1 CPU • Multiple CPU-bound threads fight each other • Could be better http://www.dabeaz.com/GIL
  98. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com A Solution 108

    • Naturally, we must reinvent the one thing that threads are good at • Namely, waiting around. • Event-loops, async, coroutines, green threads. • Think about it: These are focused on I/O (yes, I know there are other potential issues with threads, but work with me here)
  99. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com CPU-Bound Work 109

    • Event-loops have their own issues • Don't bug me, I'm blocking right now (source: chicagotribune.com)
  100. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Standard Solution 110

    • Delegate the work out to a process pool python python python python python main program CPU CPU CPU CPU • multiprocessing, concurrent.futures, etc.
  101. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 111

    • Didn't we just do this with inlined futures? def fib(n): return 1 if n <= 2 else (fib(n-1) + fib(n-2)) @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result pool = ProcessPoolExecutor(4) result = run_inline_future(compute_fibs(35)) • It runs without crashing (let's ship it!)
  102. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 112

    • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t
  103. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 113

    • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 9.56s
  104. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thought Experiment 114

    • Can you launch tasks in parallel? t1 = start_inline_future(compute_fibs(34)) t2 = start_inline_future(compute_fibs(34)) result1 = t1.result() result2 = t2.result() • Sequential execution run_inline_future(compute_fibs(34)) run_inline_future(compute_fibs(34)) 9.56s • Recall (from earlier) def start_inline_future(fut): t = Task(fut) t.step() return t 4.78s Inlined tasks running outside confines of the GIL?
  105. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Execution Model 115

    • The way in which it works is a little odd @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result add this • Output: (2 Tasks) <_MainThread(MainThread, started 140735086636224)> <_MainThread(MainThread, started 140735086636224)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> <Thread(Thread-1, started daemon 4320137216)> ... ????
  106. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Process Pools 116

    • Process pools involve a hidden result thread main thread python python CPU CPU submit result_thread • result thread reads returned values • Sets the result on the associated Future • Triggers the callback function (if any) results
  107. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com The Issue 117

    • Our inlined future switches execution threads @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result main thread result thread • Switch occurs at the first yield • All future execution occurs in result thread • That could be a little weird (especially if it blocked)
  108. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Important Lesson 118

    • If you're going to play with control flow, you must absolutely understand possible implications under the covers (i.e., switching threads across the yield statement).
  109. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Insight 119 •

    The yield is not implementation @inlined_future def compute_fibs(n): result = [] for i in range(n): val = yield from pool.submit(fib, i) result.append(val) return result • You can implement different execution models • You don't have to follow a formulaic rule
  110. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Inlined Thread Execution

    120 • Variant: Run generator entirely in a single thread def run_inline_thread(gen): value = None exc = None while True: try: if exc: fut = gen.throw(exc) else: fut = gen.send(value) try: value = fut.result() exc = None except Exception as e: exc = e except StopIteration as exc: return exc.value • It just steps through... no callback function
  111. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 121

    • Try it again with a thread pool (because why not?) @inlined_future def compute_fibs(n): result = [] for i in range(n): print(threading.current_thread()) val = yield from pool.submit(fib, i) result.append(val) return result tpool = ThreadPoolExecutor(8) t1 = tpool.submit(run_inline_thread(compute_fibs(34))) t2 = tpool.submit(run_inline_thread(compute_fibs(34))) result1 = t1.result() result2 = t2.result()
  112. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com New Execution 122

    • Output: (2 Threads) <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> <Thread(Thread-1, started 4319916032)> <Thread(Thread-2, started 4326428672)> ... (works perfectly) 4.60s (a bit faster) • Processes, threads, and futures in perfect harmony • Uh... let's move along. Faster. Must go faster.
  113. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Big Idea 123

    • You can mold and adapt generator execution • That yield statement: magic!
  114. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Actors 125 •

    There is a striking similarity between coroutines and actors (i.e., the "actor" model) • Features of Actors • Receive messages • Send messages to other actors • Create new actors • No shared state (messages only) • Can coroutines serve as actors?
  115. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 126 •

    A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 idea: use generators to define a kind of "named" actor task
  116. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Attempt 1 127

    _registry = { } def send(name, msg): _registry[name].send(msg) def actor(func): def wrapper(*args, **kwargs): gen = func(*args, **kwargs) next(gen) _registry[func.__name__] = gen return wrapper • Make a central coroutine registry and a decorator • Let's see if it works...
  117. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Example 128 •

    A very simple example @actor def printer(): while True: msg = yield print('printer:', msg) printer() n = 10 while n > 0: send('printer', n) n -=1 • It seems to work (maybe) printer: 10 printer: 9 printer: 8 ... printer: 1
  118. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 129

    • Recursive ping-pong (inspired by Stackless) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) ping pong
  119. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 130

    ping 0 pong 1 Traceback (most recent call last): File "actor.py", line 36, in <module> send('ping', 0) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 24, in ping send('pong', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) File "actor.py", line 31, in pong send('ping', n + 1) File "actor.py", line 8, in send _registry[name].send(msg) ValueError: generator already executing • Alas, it does not work
  120. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Problems 131 •

    Important differences between actors/coroutines • Concurrent execution • Asynchronous message delivery • Although coroutines have a "send()", it's a normal method call • Synchronous • Involves the call-stack • Does not allow recursion/reentrancy
  121. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 1 132

    class Actor(threading.Thread): def __init__(self, gen): super().__init__() self.daemon = True self.gen = gen self.mailbox = Queue() self.start() def send(self, msg): self.mailbox.put(msg) def run(self): while True: msg = self.mailbox.get() self.gen.send(msg) • Wrap the generator with a thread • Err...... no.
  122. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Solution 2 133

    _registry = { } _msg_queue = deque() def send(name, msg): _msg_queue.append((name, msg)) def run(): while _msg_queue: name, msg = _msg_queue.popleft() _registry[name].send(msg) • Write a tiny message scheduler • send() simply drops messages on a queue • run() executes as long as there are messages
  123. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 134

    • Recursive ping-pong (reprise) @actor def ping(): while True: n = yield print('ping %d' % n) send('pong', n + 1) @actor def pong(): while True: n = yield print('pong %d' % n) send('ping', n + 1) ping() pong() send('ping', 0) run() ping pong
  124. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Advanced Example 135

    ping 0 pong 1 ping 2 pong 3 ping 4 ping 5 ping 6 pong 7 ... ... forever • It works! • That's kind of amazing
  125. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Comments 136 •

    It's still kind of a fake actor • Lacking in true concurrency • Easily blocked • Maybe it's good enough? • I don't know • Key idea: you can bend space-time with yield
  126. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Let's Write a

    Compiler 138 • Well, an extremely simple one anyways... • Evaluating mathematical expressions 2 + 3 * 4 - 5 • Why? • Because eval() is for the weak, that's why
  127. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 139

    • Lexing : Make tokens 2 + 3 * 4 - 5 [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] • Parsing : Make a parse tree [NUM,PLUS,NUM,TIMES,NUM,MINUS,NUM] PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5)
  128. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Compilers 101 140

    • Evaluation : Walk the parse tree PLUS NUM (2) TIMES NUM (3) NUM (4) MINUS NUM (5) 9 • It's almost too simple
  129. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 141 import

    re from collections import namedtuple tokens = [ r'(?P<NUM>\d+)', r'(?P<PLUS>\+)', r'(?P<MINUS>-)', r'(?P<TIMES>\*)', r'(?P<DIVIDE>/)', r'(?P<WS>\s+)', ] master_re = re.compile('|'.join(tokens)) Token = namedtuple('Token', ['type','value']) def tokenize(text): scan = master_re.scanner(text) return (Token(m.lastgroup, m.group()) for m in iter(scan.match, None) if m.lastgroup != 'WS')
  130. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tokenizing 142 text

    = '2 + 3 * 4 - 5' for tok in tokenize(text): print(tok) Token(type='NUM', value='2') Token(type='PLUS', value='+') Token(type='NUM', value='3') Token(type='TIMES', value='*') Token(type='NUM', value='4') Token(type='MINUS', value='-') Token(type='NUM', value='5') • Example:
  131. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Parsing 143 •

    Must match the token stream against a grammar expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM • An expression is just a bunch of terms 2 + 3 * 4 - 5 term + term - term • A term is just one or more factors term term factor factor * factor (2) (3) (4)
  132. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse

    144 expr ::= term { +|- term }* term ::= factor { *|/ factor}* factor ::= NUM def expr(): ! term() ! while accept('PLUS','MINUS'): term() ! print('Matched expr') def term(): ! factor() while accept('TIMES','DIVIDE'): factor() print('Matched term') def factor(): if accept('NUM'): print('Matched factor') else: raise SyntaxError() Encode the grammar as a collection of functions Each function steps through the rule
  133. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Recursive Descent Parse

    145 def parse(toks): lookahead, current = next(toks, None), None def accept(*toktypes): nonlocal lookahead, current if lookahead and lookahead.type in toktypes: current, lookahead = lookahead, next(toks, None) return True def expr(): term() while accept('PLUS','MINUS'): term() print('Matched expr') ... expr()
  134. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 146

    class Node: _fields = [] def __init__(self, *args): for name, value in zip(self._fields, args): setattr(self, name, value) class BinOp(Node): _fields = ['op', 'left', 'right'] class Number(Node): _fields = ['value'] • Need some tree nodes for different things • Example: n1 = Number(3) n2 = Number(4) n3 = BinOp('*', n1, n2)
  135. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Tree Building 147

    def parse(toks): ... def expr(): left = term() while accept('PLUS','MINUS'): left = BinOp(current.value, left) left.right = term() return left def term(): left = factor() while accept('TIMES','DIVIDE'): left = BinOp(current.value, left) left.right = factor() return left def factor(): if accept('NUM'): return Number(int(current.value)) else: raise SyntaxError() return expr() Building nodes and hooking them together
  136. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Our Little Parser

    148 text = '2 + 3*4 - 5' toks = tokenize(text) tree = parse(toks) • Story so far... BinOp('-', BinOp('+', Number(2), BinOp('*', Number(3), Number(4) ) ), Number(5) )
  137. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 149 class

    NodeVisitor: def visit(self, node): return getattr(self, 'visit_' + type(node).__name__)(node) • The "Visitor" pattern • Example: class MyVisitor(NodeVisitor): def visit_Number(self, node): print(node.value) def visit_BinOp(self, node): self.visit(node.left) self.visit(node.right) print(node.op) MyVisitor().visit(tree) 2 3 4 * + 5 - output
  138. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 150 class

    Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator 9
  139. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Digression 151 •

    Last 12 slides a whole graduate CS course • Plus at least one additional Python tutorial • Don't worry about it • Left as an exercise...
  140. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Death Spiral 152

    # Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) • And it almost works... Traceback (most recent call last): File "compiler.py", line 100, in <module> val = Evaluator().visit(tree) File "compiler.py", line 63, in visit return getattr(self, 'visit_' + type(node).__name__)(node) File "compiler.py", line 80, in visit_BinOp leftval = self.visit(node.left) ... RuntimeError: maximum recursion depth exceeded while calling a Python object
  141. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 153 class

    Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = self.visit(node.left) rightval = self.visit(node.right) if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator !%*@*^#^# Recursion (damn you to hell)
  142. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 154 +

    0 + 1 + 2 + 3 + 4 ... + 999 0 + + + + 2 3 998 999 1 ... A deeply nested tree structure Blows up!
  143. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com 155 • The

    visitor pattern is bad idea • Better: Functional language with pattern matching and tail-call optimization I Told You So
  144. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Evaluation 158 class

    Evaluator(NodeVisitor): def visit_Number(self, node): ! return node.value def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': return leftval + rightval elif node.op == '-': return leftval - rightval elif node.op == '*': return leftval * rightval elif node.op == '/': return leftval / rightval print(Evaluator().visit(tree)) • An Expression Evaluator Nope. Not doing that recursion.
  145. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 159

    class NodeVisitor: def genvisit(self, node): result = getattr(self, 'visit_' + type(node).__name__)(node) if isinstance(result, types.GeneratorType): result = yield from result return result • Step 1: Wrap "visiting" with a generator • Thinking: No matter what the visit_() method produces, the result will be a generator • If already a generator, then just delegate to it
  146. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 160

    >>> v = Evaluator() >>> n = Number(2) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 2 >>> • Example: A method that simply returns a value • Result: Carried as value in StopIteration
  147. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 161

    >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes (iteration) Again, note the return mechanism
  148. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Generator Wrapping 162

    >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> gen = v.genvisit(n) >>> gen <generator object genvisit at 0x10070ab88> >>> gen.send(None) <__main__.Number object at 0x1058525c0> >>> gen.send(_.value) <__main__.Number object at 0x105852630> >>> gen.send(_.value) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> • A method that yields nodes def visit_Number(self, node): return node.value Manually carrying out this method in the example
  149. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 163

    class NodeVisitor: def visit(self, node): stack = [ self.genvisit(node) ] ! result = None while stack: try: node = stack[-1].send(result) stack.append(self.genvisit(node)) result = None except StopIteration as exc: ! stack.pop() result = exc.value ! return result • Step 2: Run depth-first traversal with a stack • Basically, a stack of running generators
  150. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Transcendence 164 #

    Make '0+1+2+3+4+...+999' text = '+'.join(str(x) for x in range(1000)) toks = tokenize(text) tree = parse(toks) val = Evaluate().visit(tree) print(val) • Does it work? • Yep 499500 • Yow!
  151. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 165

    class Evaluator(NodeVisitor): def visit_BinOp(self, node): leftval = yield node.left rightval = yield node.right if node.op == '+': result = leftval + rightval ... return result • Each yield creates a new stack entry • Returned values (via StopIteration) get propagated as results generator generator generator stack yield yield return return
  152. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 166

    >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Nodes are visited and generators pushed onto a stack
  153. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Running Recursion 167

    >>> v = Evaluator() >>> n = BinOp('*', Number(3), Number(4)) >>> stack = [ v.genvisit(n) ] >>> stack[-1].send(None) <__main__.Number object at 0x1058525c0> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 3 >>> stack.pop() >>> stack[-1].send(3) <__main__.Number object at 0x105852630> >>> stack.append(v.genvisit(_)) >>> stack[-1].send(None) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 4 >>> stack.pop() >>> stack[-1].send(4) Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration: 12 >>> Results propagate via StopIteration 12 (Final Result)
  154. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Historical Perspective 169

    • Generators seem to have started as a simple way to implement iteration (Python 2.3) • Took an interesting turn with support for coroutines (Python 2.5) • Taken to a whole new level with delegation support in PEP-380 (Python 3.3).
  155. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Control Flow Bending

    170 • yield statement allows you to bend control-flow to adapt it to certain kinds of problems • Wrappers (context managers) • Futures/concurrency • Messaging • Recursion • Frankly, it blows my mind.
  156. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com asyncio 171 •

    Inclusion of asyncio in standard library may be a game changer • To my knowledge, it's the only standard library module that uses coroutines/generator delegation in a significant manner • To really understand how it works, need to have your head wrapped around generators • Read the source for deep insight
  157. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Is it Proper?

    172 • Are coroutines/generators a good idea or not? • Answer: I still don't know • Issue: Coroutines seem like they're "all in" • Fraught with potential mind-bending issues • Example: Will there be two standard libraries?
  158. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Two Libraries? 173

    Python Standard Library Standard coroutine library (asyncio and friends) ????? • If two different worlds, do they interact? • If so, by what rules?
  159. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Personal Use 174

    • My own code is dreadfully boring • Generators for iteration: Yes. • Everything else: Threads, recursion, etc. (sorry) • Nevertheless: There may be something to all of this advanced coroutine/generator business
  160. Copyright (C) 2014, David Beazley (@dabeaz). http://www.dabeaz.com Thanks! 176 •

    I hope you got some new ideas • Please feel free to contact me http://www.dabeaz.com • Also, I teach Python classes (shameless plug) @dabeaz (Twitter) • Special Thanks: Brian Curtin, Ken Izzo, George Kappel, Christian Long, Michael Prentiss, Vladimir Urazov, Guido van Rossum