Code Generation in Python — Dismantling Jinja

A talk I gave at PyCon 2012 about code generation and how Jinja2 works internally.

Transcript

1. Code Generation in Python Dismantling Jinja a
2. bit.ly/codegeneration Discuss this presentation, give feedback
3. Code Generation?
4. eval is evil Or is it?
5. Why is eval evil? Security Performance &
6. Security Code Injection Namespace pollution
7. Performance No bytecode Code makes code that code runs
8. So: Why? No suitable alternatives
9. use responsibly because of this:
10. 101 EVAl
11. >>> code = compile('a = 1 + 2', '<string>', 'exec') >>> code <code object <module> at 0x1004d5120, file "<string>", line 1> Compile
12. >>> ns = {} >>> exec code in ns >>> ns['a'] 3 Eval
13. >>> import ast >>> ast.parse('a = 1 + 2') <_ast.Module object at 0x1004fd250> >>> code = compile(_, '<string>', 'exec') AST #1
14. AST #2 >>> n = ast.Module([ ... ast.Assign([ast.Name('a', ast.Store())], ... ast.BinOp(ast.Num(1), ast.Add(), ... ast.Num(2)))])) >>> ast.fix_missing_locations(n) >>> code = compile(n, '<string>', 'exec')
15. No strings passed to eval()/exec Explicit compilation to bytecode Execution in explicit namespace Recap
16. ARCHITECTURE TeMpLAtE eNgInE
18. 2nd Iteration Generates Python Code Python Semantics Different Scoping Overview
19. Lexer Pipeline Parser Identifier Analyzer Code Generator Python Source Bytecode Runtime
20. Different Scoping WSGI & Generating Debug-ability Restricted Environments Complexities
21. <ul> {% for item in seq %} <li>{{ item }}</li> {% endfor %} </ul> Input
22. print "<ul>" for each item in the variable seq push the scope print "<li>" print the value of item and escape it as necessary print "</li>" pop the scope print "</ul>" Behavior
23. Naive: write(u'<ul>') for _tmp in context['seq']: context.push({'item': _tmp}) write(u'<li>') write(autoescape(context['item'])) write(u'</li>') context.pop() write(u'</ul>')
24. Actual: l_seq = context.resolve('seq') write(u'<ul>') for l_item in l_seq: write(u'<li>') write(autoescape(l_item)) write(u'</li>') write(u'</ul>')
25. ?
26. COMPILATION INtRoDUCTIoN tO
27. Low Level High Level e Art of Code Generation versus
28. Low Level Code Generation 2 0 LOAD_CONST 1 (1) 3 LOAD_CONST 2 (2) 6 BINARY_ADD 7 STORE_FAST 0 (a) a = 1 + 2
29. Assign(targets=[Name(id='a', ctx=Store())], value=BinOp(left=Num(n=1), op=Add(), right=Num(n=2)))] a = 1 + 2 High Level Code Generation
30. Bytecode Abstract Syntax Trees Sourcecode Building Blocks
31. Bytecode Undocumented Does not work on GAE Implementation Specific
32. AST More Limited Easier to Debug Does not segfault the Interpreter
33. Source Works always Very Limited Hard to Debug without Hacks
34. e Tale of Two Pieces of Code (very similar pieces of code)
35. def foo(): a = 0 for x in xrange(100): a += x print a foo() Fast
36. a = 0 for x in xrange(100): a += x print a Slower
37. ?
38. 2 0 LOAD_CONST 0 (0) 3 STORE_NAME 0 (a) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_NAME 1 (xrange) 12 LOAD_CONST 1 (100) 15 CALL_FUNCTION 1 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_NAME 2 (x) 4 25 LOAD_NAME 0 (a) 28 LOAD_NAME 2 (x) 31 INPLACE_ADD 32 STORE_NAME 0 (a) 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_NAME 0 (a) 42 PRINT_ITEM 43 PRINT_NEWLINE Slower
39. 2 0 LOAD_CONST 1 (0) 3 STORE_FAST 0 (a) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_GLOBAL 0 (xrange) 12 LOAD_CONST 2 (100) 15 CALL_FUNCTION 1 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_FAST 1 (x) 4 25 LOAD_FAST 0 (a) 28 LOAD_FAST 1 (x) 31 INPLACE_ADD 32 STORE_FAST 0 (a) 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_FAST 0 (a) 42 PRINT_ITEM 43 PRINT_NEWLINE Fast
40. 2 0 LOAD_CONST 1 (0) 3 STORE_FAST 0 (a) 3 6 SETUP_LOOP 30 (to 39) 9 LOAD_GLOBAL 0 (xrange) 12 LOAD_CONST 2 (100) 15 CALL_FUNCTION 1 18 GET_ITER >> 19 FOR_ITER 16 (to 38) 22 STORE_FAST 1 (x) 4 25 LOAD_FAST 0 (a) 28 LOAD_FAST 1 (x) 31 INPLACE_ADD 32 STORE_FAST 0 (a) 35 JUMP_ABSOLUTE 19 >> 38 POP_BLOCK 5 >> 39 LOAD_FAST 0 (a) 42 PRINT_ITEM 43 PRINT_NEWLINE Fast
41. >>> def foo(): ... a = 42 ... locals()['a'] = 23 ... return a ... >>> foo() 42 Example
42. SEMANTICS A StOrY ABoUT
43. print "<ul>" for each item in the variable seq push the scope print "<li>" print the value of item and escape it as necessary print "</li>" pop the scope print "</ul>" Remember
44. at's not how Python works … so how do you generate code for it?
45. Keep tracks of identifiers emulate desired semantics Tracking
46. Context in Jinja2 is a Data Source Context in Django is a Data Store Scopes
47. <ul> {% for item in seq %} {% include "item.html" %} {% endfor %} </ul> Source
48. Code l_seq = context.resolve('seq') write(u'<ul>') for l_item in l_seq: t1 = env.get_template('other.html') for event in yield_from(t1, context, {'item': l_item}) yield event write(u'</ul>')
49. What happens in the include … … stays in the include
50. Impossible @contextfunction def get_users_and_store(context, var='users'): context[var] = get_all_users() return u''
51. EXAMPLES PrACTICAl
52. Source <ul class=navigation> {% for item in sequence %} <li>{{ item }}</li> {% endfor %} </ul>
53. Generated def root(context): l_sequence = context.resolve('sequence') yield u'\n<ul class=navigation>\n' l_item = missing for l_item in l_sequence: yield u'\n <li>%s</li>' % ( escape(l_item), ) l_item = missing yield u'\n</ul>'
54. Source <ul class=navigation> {% for item in sequence %} <li>{{ loop.index }}: {{ item }}</li> {% endfor %} </ul>
55. Generated def root(context): l_sequence = context.resolve('sequence') yield u'\n<ul class=navigation>\n' l_item = missing for l_item, l_loop in LoopContext(l_sequence): yield u'\n <li>%s: %s</li>\n' % ( escape(environment.getattr(l_loop, 'index')), escape(l_item), ) l_item = missing yield u'\n</ul>'
56. Source <ul class=navigation> {% for item in sequence %} <li>{{ loop.index }}: {{ item }}</li> {% endfor %} </ul> <p>Item: {{ item }}
57. Generated def root(context): l_item = context.resolve('item') l_sequence = context.resolve('sequence') yield u'\n<ul class=navigation>\n' t_1 = l_item for l_item, l_loop in LoopContext(l_sequence): yield u'\n <li>%s: %s</li>\n' % ( escape(environment.getattr(l_loop, 'index')), escape(l_item), ) l_item = t_1 yield u'\n</ul>\n<p>Item: ' yield escape(l_item)
58. Source {% extends "layout.html" %} {% block body %} <h1>Hello World!</h1> {% endblock %}
59. Generated def root(context): parent_template = environment.get_template('layout.html', None) for name, parent_block in parent_template.blocks.iteritems(): context.blocks.setdefault(name, []).append(parent_block) for event in parent_template.root_render_func(context): yield event def block_body(context): if 0: yield None yield u'\n <h1>Hello World!</h1>\n' blocks = {'body': block_body}
60. Source <!doctype html> {% block body %}{% endblock %}
61. Generated def root(context): yield u'<!doctype html>\n' for event in context.blocks['body'][0](context): yield event def block_body(context): if 0: yield None blocks = {'body': block_body}
62. Source {% extends "layout.html" %} {% block title %}Hello | {{ super() }}{% endblock %}
63. Generated def root(context): parent_template = environment.get_template('layout.html', None) for name, parent_block in parent_template.blocks.iteritems(): context.blocks.setdefault(name, []).append(parent_block) for event in parent_template.root_render_func(context): yield event def block_title(context): l_super = context.super('title', block_title) yield u'Hello | ' yield escape(context.call(l_super)) blocks = {'title': block_title}
64. Jinja Do WHY DOeS
65. … manual code generation? why Originally the only option AST compilation was new in 2.6 GAE traditionally did not allow it
66. … generators instead of buffer.append() why Required for WSGI streaming unless greenlets are in use Downside: StopIteration :-(
67. … map "var_x" to "l_var_x" why Reversible to debugging purposes Does not clash with internals see templatetk for better approach
68. Jinja Do HOW DoEs
69. … does automatic escaping work how Markup object Operator overloading Compile-time and Runtime
70. Const <h1>{{ "<strong>Hello World!</strong>" }}</h1> def root(context): yield u'<h1>&lt;strong&gt;Hello World!&lt;/strong&gt;</h1>'
71. Runtime <h1>{{ variable }}</h1> def root(context): l_variable = context.resolve('variable') yield u'<h1>%s</h1>' % ( escape(l_variable), )
72. Control #1 {% autoescape false %}<h1>{{ variable }}</h1>{% endautoescape %} def root(context): l_variable = context.resolve('variable') t_1 = context.eval_ctx.save() context.eval_ctx.autoescape = False yield u'<h1>%s</h1>' % ( l_variable, ) context.eval_ctx.revert(t_1)
73. Control #2 {% autoescape flag %}<h1>{{ variable }}</h1>{% endautoescape %} def root(context): l_variable = context.resolve('variable') l_flag = context.resolve('flag') t_1 = context.eval_ctx.save() context.eval_ctx.autoescape = l_flag yield u'%s%s%s' % ( (context.eval_ctx.autoescape and escape or to_string)((context.eval_ctx.autoescape and Markup or identity)(u'<h1>')), (context.eval_ctx.autoescape and escape or to_string)(l_variable), (context.eval_ctx.autoescape and escape or to_string)((context.eval_ctx.autoescape and Markup or identity)(u'</h1>')), ) context.eval_ctx.revert(t_1)
74. … far does the Markup object go? how All operators are overloaded All string operations are safe necessary due to operator support
75. Example >>> from markupsafe import Markup >>> Markup('<em>%s</em>') % '<insecure>' Markup(u'<em>&lt;insecure&gt;</em>') >>> Markup('<em>') + '<insecure>' + Markup('</em>') Markup(u'<em>&lt;insecure&gt;</em>') >>> Markup('<em>Complex&nbsp;value</em>').striptags() u'Complex\xa0value'
76. … do unde ned values work how Configurable Replaced by special object By default one level of silence
77. Example >>> from jinja2 import Undefined >>> unicode(Undefined(name='missing_var')) u'' >>> unicode(Undefined(name='missing_var').attribute) Traceback (most recent call last): File "<console>", line 1, in <module> UndefinedError: 'missing_var' is undefined
78. Q&A
79. @mitsuhiko http://lucumr.pocoo.org/ armin.ronacher@active-4.com
80. Oh hai. We're hiring http://fireteam.net/careers