Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Elizaveta Shashkova - Debugging in Python 3.6: ...

Elizaveta Shashkova - Debugging in Python 3.6: Better, Faster, Stronger

Python 3.6 was released in December of 2016 and it has a lot of new cool features. Some of them are quite easy for using: a developer can read, for example, about f-strings and they can start using them in their programs as soon as possible. But sometimes features are not so evident, and a new frame evaluation API is one of them.
The new frame evaluation API was introduced to CPython in PEP 523 and it allows to specify a per-interpreter function pointer to handle the evaluation of frames. It might not be evident how to use this new feature in everyday life, but it’s quite easy to understand how to build a fast debugger based on it.
In this talk we are going to explain how standard way of debugging in Python works and how a new frame evaluation API may be useful for creating the fast debugger. Also we will consider why such fast debugging was not possible in the previous versions of Python. If someone hasn’t made a final decision to move to Python 3.6 this talk will provide some new reasons to do it.

https://us.pycon.org/2017/schedule/presentation/526/

PyCon 2017

May 21, 2017
Tweet

More Decks by PyCon 2017

Other Decks in Programming

Transcript

  1. Bio • Software developer of PyCharm IDE at JetBrains •

    Debugger • Saint Petersburg, Russia 2
  2. 8 def tracefunc(frame, event, arg): print(frame.f_lineno, event) return tracefunc sys.settrace(tracefunc)

    1 2 3 4 5 6 sys.settrace(tracefunc) - set the system tracing function Tracing Function
  3. def foo(): friends = ["Bob", "Tom"] for f in friends:

    print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 Tracing Function 9
  4. 1 2 3 4 5 6 7 8 9 1

    call def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 10
  5. 1 2 3 4 5 6 7 8 9 1

    call 2 line def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 11
  6. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 12
  7. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 13
  8. 1 2 3 4 5 6 7 8 9 1

    call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return def foo(): friends = ["Bob", "Tom"] for f in friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() Tracing Function 14
  9. Performance def foo(): friends = ["Bob", "Tom"] for f in

    friends: print("Hi %s!” % f) return len(friends) sys.settrace(tracefunc) foo() 1 2 3 4 5 6 7 8 9 1 call 2 line 3 line 4 line Hi Bob! 3 line 4 line Hi Tom! 5 line 5 return 17
  10. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum 1 2 3 4 5 6 7 8 9 18
  11. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum def tracefunc(frame, event, arg): return tracefunc 1 2 3 4 5 6 7 8 9 19
  12. Frame Evaluation def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number =

    frame.f_lineno print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 30
  13. def frame_eval(frame, exc): func_name = frame.f_code.co_name a line_number = frame.f_lineno

    print(line_number, func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 31
  14. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 32
  15. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 33
  16. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) 1 2 3 4 5 6 7 8 9 Frame Evaluation 34
  17. def frame_eval(frame, exc): func_name = frame.f_code.co_name line_number = frame.f_lineno print(line_number,

    func_name) return _PyEval_EvalFrameDefault(frame, exc) def set_frame_eval(): state = PyThreadState_Get() state.interp.eval_frame = frame_eval 1 2 3 4 5 6 7 8 9 Frame Evaluation 35
  18. Example 1 2 3 4 5 6 7 8 9

    10 11 def first(): second() def second(): third() def third(): pass set_frame_eval() first() 36
  19. Example def first(): second() def second(): third() def third(): pass

    set_frame_eval() first() 1 2 3 4 5 6 7 8 9 10 11 1 first 4 second 7 third 37
  20. Custom Frame Evaluation • It works! • Executed while entering

    a frame • Access to frame and code object 38
  21. Breakpoints • Access to the whole code object • Insert

    breakpoint’s code into frame’s code 46
  22. Breakpoints def maximum(a, b): if a > b: return a

    else: return b 1 2 3 4 5 6 7 8 47
  23. Breakpoints def maximum(a, b): if a > b: return a

    # breakpoint else: return b 1 2 3 4 5 6 7 8 48
  24. Breakpoints def maximum(a, b): if a > b: return a

    # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 49
  25. Breakpoints def maximum(a, b): if a > b: breakpoint() return

    a # breakpoint else: return b 1 2 3 4 5 6 7 8 50
  26. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 51
  27. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 52
  28. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 53
  29. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 54
  30. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 55
  31. Python Bytecode def maximum(a, b): if a > b: return

    a else: return b import dis dis.dis(maximum) 1 2 3 4 5 6 7 8 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 56
  32. Python Bytecode 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE 57
  33. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 58
  34. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 59
  35. 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1 (b) 4

    COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE Python Bytecode 60
  36. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() 61
  37. Bytecode Modification 2 0 LOAD_FAST 0 (a) 2 LOAD_FAST 1

    (b) 4 COMPARE_OP 4 (>) 6 POP_JUMP_IF_FALSE 12 3 8 LOAD_FAST 0 (a) 10 RETURN_VALUE 5 >> 12 LOAD_FAST 1 (b) 14 RETURN_VALUE 16 LOAD_CONST 0 (None) 18 RETURN_VALUE breakpoint() ?! 64
  38. Breakpoint Bytecode def _stop_at_break(): # a lot of code here

    def breakpoint(): _stop_at_break() 1 2 3 4 5 6 7 8 0 LOAD_GLOBAL 0 2 CALL_FUNCTION 0 4 POP_TOP 6 LOAD_CONST 0 8 RETURN_VALUE 65
  39. Example 1 def calculate(): sum = 0 for i in

    range(10 ** 7): sum += i return sum 1 2 3 4 5 69
  40. Example 2 def foo(): pass def calculate(): sum = 0

    for i in range(10 ** 7): foo() sum += i return sum 1 2 3 4 5 6 7 8 9 72
  41. PEP 523 • Expand PyCodeObject struct • co_extra - “scratch

    space” for the code object • Mark frames without breakpoints 75
  42. Mark Frames def frame_eval(frame, exc): flag = _PyCode_GetExtra(frame.f_code, index) if

    flag == NO_BREAKS_IN_FRAME: return _PyEval_EvalFrameDefault(frame, exc)
 
 # check for breakpoints ... 1 2 3 4 5 6 7 8 9 76
  43. Use cases def maximum(a, b): if a > b: return

    a # breakpoint else: return b 1 2 3 4 5 6 7 8 breakpoint() 88