Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Exploring our Python Interpreter

Exploring our Python Interpreter

Presented at
* PyCon Ireland 2015
* PyCon Canada 2015
* PythonFOSDEM 2016
* EuroPython 2016

Stéphane Wirtel

November 07, 2015
Tweet

More Decks by Stéphane Wirtel

Other Decks in Programming

Transcript

  1. I am ... Stephane Wirtel a #python lover not a

    CPython Core Dev just a contributor of CPython, Gunicorn fellow @thePSF member of the EuroPython Society 1 / 49
  2. Reminder It's just an introduction I am not a CPython

    Core Dev, just a contributor 2 / 49
  3. Schedule How to start with CPython ? What's the result

    of python -c "2 + 2"? Summary 3 / 49
  4. Start with - DevGuide Developer's Guide Comprehensive resource for contributing

    to Python Quick Start Grammar Design of the CPython Compiler ... https://docs.python.org/devguide 5 / 49
  5. Start with - DevGuide - Mentorship Mentors Provide an open

    and welcoming place to connect students, programmers and anyone interested in contributing to the Python Core development. Try to maintain contributor interest One mailing list Mentors: Guido Van Rossum Brett Cannon David R. Murray Victor Stinner ... myself ... ;-) ... http://www.pythonmentors.com 6 / 49
  6. Start with - DevGuide - Mentorship - Mailing Lists The

    Mailing Lists Python Bugs Announce: ML/new-bugs-announce Python Bugs: ML/python-bugs-list Python Core Mentorship: ML/core-mentorship Python Dev: ML/python-dev Python Ideas: ML/python-ideas FAT Python Python Speed: ML/speed ML = https://mail.python.org/mailman/listinfo 7 / 49
  7. Start with - DevGuide - Mentorship - Mailing Lists -

    Contributing Step 1 Create a user account on https://bugs.python.org Sign a Contributor Agreement after that, you can ... 8 / 49
  8. Start with - DevGuide - Mentorship - Mailing Lists -

    Contributing Step 2 Improve the docs Fill Issues if you find a bug Test the proposed patches (Review) Write Patches (Fixes, features) be stressed and now, wait 6 months for a review ;-) BUT the process is really slow and laborious :/ We need reviewers and developers FYI: Migrating from hg.python.org to Github PEP512 9 / 49
  9. Overview of the directories . ├── D o c =

    = > t h e F i n e M a n u a l . . . ├── G r a m m a r = = > W h e r e t h e g r a m m a r i s d e f i n e d ├── I n c l u d e = = > t h e C H e a d e r s ├── L i b = = > t h e . p y m o d u l e s ├── M o d u l e s = = > t h e . c m o d u l e s ├── O b j e c t s = = > t h e b u i l t i n o b j e c t s ( b o o l , d i c t , s e t , t u p l e , l i s t , u n i c o d e . . . ) ├── P a r s e r = = > g r a m m a r , l e x e r , p a r s e r , c o m p i l e r ├── P r o g r a m s = = > t h e p y t h o n e x e c u t a b l e └── P y t h o n = = > t h e v i r t u a l m a c h i n e 12 / 49
  10. About the documentation Do YOU want to contribute ??? here

    is a small idea: We need a tutorial on AsyncIO and how to use it because the documentation of AsyncIO is useful but it's not a tutorial but a REFERENCE. 14 / 49
  11. CLI Command Line # T h e P y t

    h o n e x e c u t a b l e P r o g r a m / p y t h o n . c : m a i n # T h e P y t h o n l i b r a r y - > M o d u l e s / m a i n . c : P y _ M a i n # I n i t i a l i z e C P y t h o n - > P y I n i t i a l i z e - > P y R u n _ S i m p l e S t r i n g F l a g s - > P y I m p o r t _ A d d M o d u l e - > P y R u n _ S t r i n g F l a g s - > P y P a r s e r _ A S T F r o m S t r i n g O b j e c t - > P y P a r s e r _ P a r s e S t r i n g O b j e c t - > P y A S T _ C o m p i l e O b j e c t - > P y E v a l _ E v a l C o d e 21 / 49
  12. CLI Parsing - Lexer Lexer t o k e n

    s = l e x e r ( ' x = 2 + 2 ' ) > > > s t a t e m e n t = ' x = 2 + 2 ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n i z e . g e n e r a t e _ t o k e n s ( t o k e n s ) ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' x ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' + ' ) , T o k e n I n f o ( t y p e = N U M B E R , s t r i n g = ' 2 ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) , ] in P a r s e r / t o k e n i z e r . c -> P y T o k e n i z e r _ F r o m S t r i n g in P a r s e r / p a r s e t o k . c -> p a r s e t o k in L i b / t o k e n i z e . p y 22 / 49
  13. CLI Parsing - Lexer Lexer Mysterious async keyword in Python

    3.5 > > > i m p o r t k e y w o r d > > > ' a s y n c ' i n k e y w o r d . k w l i s t F a l s e Proof > > > s t a t e m e n t = ' a s y n c = T r u e ' > > > s t r I O = i o . S t r i n g I O ( s t a t e m e n t ) . r e a d l i n e > > > t o k e n s = t o k e n i z e . g e n e r a t e _ t o k e n s ( s t r I O ) > > > p p ( l i s t ( t o k e n s ) ) [ T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' a s y n c ' ) , T o k e n I n f o ( t y p e = O P , s t r i n g = ' = ' ) , T o k e n I n f o ( t y p e = N A M E , s t r i n g = ' T r u e ' ) , T o k e n I n f o ( t y p e = E N D M A R K E R , s t r i n g = ' ' ) ] 23 / 49
  14. CLI Parsing - Lexer - Parser Parser a s t

    = p a r s e r ( t o k e n s ) > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > a s t . d u m p ( t r e e ) M o d u l e ( b o d y = [ A s s i g n ( t a r g e t s = [ N a m e ( i d = ' x ' ) ] , v a l u e = B i n O p ( l e f t = N u m ( n = 2 ) , o p = A d d ( ) , r i g h t = N u m ( n = 2 ) ) ) ] ) P y t h o n / p y t h o n r u n . c -> P y P a r s e r _ A S T F r o m S t r i n g O b j e c t 24 / 49
  15. CLI Parsing Compiler - Compiler Compiler b y t e

    c o d e s = c o m p i l e r ( a s t ) > > > i m p o r t d i s > > > i m p o r t a s t > > > t r e e = a s t . p a r s e ( ' x = 2 + 2 ' ) > > > c o d e _ o b j e c t = c o m p i l e ( t r e e , ' d u m m y . p y ' , m o d e = ' e x e c ' ) > > > d i s . d i s ( c o d e _ o b j e c t ) 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E See PEP-0339 in P y t h o n / c o m p i l e . c -> P y A S T _ C o m p i l e O b j e c t 25 / 49
  16. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes compact numeric

    codes portable code one byte followed by optional parameters used by a software interpreter # d e f i n e P O P _ T O P 1 # d e f i n e R O T _ T W O 2 . . . # d e f i n e B I N A R Y _ A D D 2 3 # d e f i n e B I N A R Y _ S U B S T R A C T 2 4 . . . # d e f i n e S E T U P _ A S Y N C _ W I T H 1 5 4 # d e f i n e F O R M A T _ V A L U E 1 5 5 See in I n c l u d e / o p c o d e . h 26 / 49
  17. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes in /

    t m p / e m p t y . p y # ! / u s r / b i n / e n v p y t h o n # a s u p e r c o m m e n t Here is the bytecode 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 27 / 49
  18. CLI Parsing Compiler - Compiler - Bytecodes Bytecodes > >

    > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 28 / 49
  19. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer

    b y t e c o d e s = o p t i m i z e r ( b y t e c o d e s ) performs basic peephole optimizations removes the dead branches in P y t h o n / p e e p h o l e . c -> P y C o d e _ O p t i m i z e 29 / 49
  20. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer

    x = 2 + 2 1 0 L O A D _ C O N S T 2 ( 4 ) 3 S T O R E _ N A M E 0 ( x ) 6 L O A D _ C O N S T 1 ( N o n e ) 9 R E T U R N _ V A L U E 30 / 49
  21. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer

    i f 1 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ N A M E 0 ( p r i n t ) 3 L O A D _ C O N S T 0 ( ' h e l l o ' ) 6 C A L L _ F U N C T I O N 1 ( 1 p o s i t i o n a l , 0 k e y w o r d p a i r ) 9 P O P _ T O P 1 0 L O A D _ C O N S T 1 ( N o n e ) 1 3 R E T U R N _ V A L U E 31 / 49
  22. CLI Parsing Compiler - Compiler - Bytecodes - Peepholer Peepholer

    i f 0 : p r i n t ( ' h e l l o ' ) 1 0 L O A D _ C O N S T 0 ( N o n e ) 3 R E T U R N _ V A L U E 32 / 49
  23. CLI Parsing Compiler Interpreter Interpreter e x e c u

    t i o n = i n t e r p r e t e r ( b y t e c o d e s ) Virtual Stack Machine Execute the ByteCode 33 / 49
  24. CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine c

    l a s s I n t e r p r e t e r : d e f _ _ i n i t _ _ ( s e l f , i n s t r u c t i o n s ) : s e l f . i n s t r u c t i o n s = i n s t r u c t i o n s s e l f . s t a c k = [ ] s e l f . i n s t _ p t r = 0 s e l f . r u n n i n g = T r u e d e f r u n ( s e l f ) : w h i l e s e l f . r u n n i n g : s e l f . e v a l ( s e l f . f e t c h ( ) ) s e l f . i n s t _ p t r + = 1 d e f f e t c h ( s e l f ) : r e t u r n s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] d e f e v a l ( s e l f , i n s t r u c t i o n ) : g e t a t t r ( s e l f , ' _ e v a l _ { } ' . f o r m a t ( i n s t r u c t i o n ) ) ( ) 34 / 49
  25. CLI Parsing Compiler Interpreter Interpreter a Virtual Stack Machine d

    e f _ e v a l _ P U S H ( s e l f ) : s e l f . i n s t _ p t r + = 1 s e l f . s t a c k . a p p e n d ( s e l f . i n s t r u c t i o n s [ s e l f . i n s t _ p t r ] ) d e f _ e v a l _ A D D ( s e l f ) : l e f t = s e l f . s t a c k . p o p ( ) r i g h t = s e l f . s t a c k . p o p ( ) r e s u l t = l e f t + r i g h t s e l f . s t a c k . a p p e n d ( r e s u l t ) d e f _ e v a l _ P O P ( s e l f ) : v a l u e = s e l f . s t a c k . p o p ( ) p r i n t ( " P o p p e d % r " % v a l u e ) d e f _ e v a l _ H A L T ( s e l f ) : s e l f . r u n n i n g = F a l s e 35 / 49
  26. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 36 / 49
  27. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    - - > P U S H 5 | - - - - - - - - - - - | P U S H 3 | 5 | P U S H 1 0 | - - - - - - - - - - - | A D D A D D P O P 37 / 49
  28. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | - - > P U S H 3 | 3 | P U S H 1 0 | - - - - - - - - - - - | A D D | 5 | A D D | - - - - - - - - - - - | P O P 38 / 49
  29. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 0 | - - > P U S H 1 0 | - - - - - - - - - - - | A D D | 3 | A D D | - - - - - - - - - - - | P O P | 5 | | - - - - - - - - - - - | 39 / 49
  30. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 3 | P U S H 1 0 | - - - - - - - - - - - | - - > A D D | 5 | A D D | - - - - - - - - - - - | P O P 40 / 49
  31. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | P U S H 3 | 1 8 | P U S H 1 0 | - - - - - - - - - - - | A D D - - > A D D P O P 41 / 49
  32. CLI Parsing Compiler Interpreter Interpreter Execution of my custom Bytecode

    P U S H 5 | - - - - - - - - - - - | P U S H 3 | | P U S H 1 0 | - - - - - - - - - - - | A D D A D D - - > P O P 42 / 49
  33. CLI Parsing Compiler Interpreter Interpreter Do you remember this function

    ? > > > d e f f u n c ( x ) : . . . x = x + 1 . . . r e t u r n x . . . And the Bytecode ? > > > d i s . d i s ( f u n c ) 2 0 L O A D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) 6 B I N A R Y _ A D D 7 S T O R E _ F A S T 0 ( x ) 3 1 0 L O A D _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E 43 / 49
  34. CLI Parsing Compiler Interpreter Interpreter 2 0 L O A

    D _ F A S T 0 ( x ) 3 L O A D _ C O N S T 1 ( 1 ) T A R G E T ( L O A D _ F A S T ) { P y O b j e c t * v a l u e = G E T L O C A L ( o p a r g ) ; i f ( v a l u e = = N U L L ) { f o r m a t _ e x c _ c h e c k _ a r g ( P y E x c _ U n b o u n d L o c a l E r r o r , U N B O U N D L O C A L _ E R R O R _ M S G , P y T u p l e _ G e t I t e m ( c o - > c o _ v a r n a m e s , o p a r g ) ) ; g o t o e r r o r ; } P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( L O A D _ C O N S T ) { P y O b j e c t * v a l u e = G E T I T E M ( c o n s t s , o p a r g ) ; P y _ I N C R E F ( v a l u e ) ; P U S H ( v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } 44 / 49
  35. CLI Parsing Compiler Interpreter Interpreter 6 B I N A

    R Y _ A D D T A R G E T ( B I N A R Y _ A D D ) { P y O b j e c t * r i g h t = P O P ( ) ; P y O b j e c t * l e f t = T O P ( ) ; P y O b j e c t * s u m ; i f ( P y U n i c o d e _ C h e c k E x a c t ( l e f t ) & & P y U n i c o d e _ C h e c k E x a c t ( r i g h t ) ) { s u m = u n i c o d e _ c o n c a t e n a t e ( l e f t , r i g h t , f , n e x t _ i n s t r ) ; } e l s e { s u m = P y N u m b e r _ A d d ( l e f t , r i g h t ) ; P y _ D E C R E F ( l e f t ) ; } P y _ D E C R E F ( r i g h t ) ; S E T _ T O P ( s u m ) ; i f ( s u m = = N U L L ) g o t o e r r o r ; D I S P A T C H ( ) ; } 45 / 49
  36. CLI Parsing Compiler Interpreter Interpreter 7 S T O R

    E _ F A S T 0 ( x ) 1 3 R E T U R N _ V A L U E T A R G E T ( S T O R E _ F A S T ) { P y O b j e c t * v a l u e = P O P ( ) ; S E T L O C A L ( o p a r g , v a l u e ) ; F A S T _ D I S P A T C H ( ) ; } T A R G E T ( R E T U R N _ V A L U E ) { r e t v a l = P O P ( ) ; w h y = W H Y _ R E T U R N ; g o t o f a s t _ b l o c k _ e n d ; } 46 / 49
  37. CLI Parsing Compiler Interpreter Interpreter And just for the fun,

    Everything is in P y t h o n / c e v a l . c but especially in one function P y E v a l _ E v a l F r a m e E x 7 9 6 P y O b j e c t * 7 9 7 P y E v a l _ E v a l F r a m e E x ( P y F r a m e O b j e c t * f , i n t t h r o w f l a g ) 7 9 8 { . . . 1 8 9 9 # i f d e f C A S E _ T O O _ B I G 1 9 0 0 d e f a u l t : s w i t c h ( o p c o d e ) { 1 9 0 1 # e n d i f . . . 3 6 4 4 } 47 / 49