of Earthly Delights", c. 2015 Pythonic • PyPI? • PyCON? Everyone is naked, riding around on exotic animals, eating giant berries, etc. ???????????? Pythonic
of Earthly Delights", c. 2015 Pythonic • PyCON? Background: I've been using Python since version 1.3. Basic use of modules and packages is second nature. However, I also realize that I don't know that much about what's happening under the covers. I want to correct that. Pythonic
of Earthly Delights", c. 2015 Pythonic • This tutorial! • PyCON? • A fresh take on modules • Goal is to reintroduce the topic • Avoid crazy hacks? (maybe) Pythonic
of Earthly Delights", c. 2015 Pythonic • PyCON? • Target Audience: Myself! • Understanding import is useful • Also: Book writing • Will look at some low level details, but keep in the mind the goal is to gain a better idea of how everything works and holds together Pythonic
of Earthly Delights", c. 2015 Pythonic • PyCON? • Perspective: I'm looking at this topic from the point of view of an application developer and how I might use the knowledge to my advantage • I am not a Python core developer • Target audience is not core devs Pythonic
of Earthly Delights", c. 2015 Pythonic • PyCON? Pythonic It's not "Modules and Packaging" The tutorial is not about package managers (setuptools, pip, etc.) ... because "reasons"
• I learned a lot preparing • Also fractured a rib while riding my bike on this frozen lake • Behold the pain killers that proved to be helpful in finishing • Er... let's start....
Python source file is a module 17 # spam.py def grok(x): ... def blah(x): ... • You use import to execute and access it import spam a = spam.grok('hello') from spam import grok a = grok('hello')
module is its own isolated world 18 # spam.py x = 42 def blah(): print(x) • What happens in a module, stays in a module These definitions of x are different # eggs.py x = 37 def foo(): print(x)
When a module is imported, all of the statements in the module execute one after another until the end of the file is reached • The contents of the module namespace are all of the global names that are still defined at the end of the execution process • If there are scripting statements that carry out tasks in the global scope (printing, creating files, etc.), you will see them run on import 20
• Lifts selected symbols out of a module after importing it and makes them available locally from math import sin, cos def rectangular(r, theta): x = r * cos(theta) y = r * sin(theta) return x, y 21 • Allows parts of a module to be used without having to type the module prefix
* • Takes all symbols from a module and places them into local scope from math import * def rectangular(r, theta): x = r * cos(theta) y = r * sin(theta) return x, y 22 • Sometimes useful • Usually considered bad style (try to avoid)
on import do not change the way that modules work 23 import math as m from math import cos, sin from math import * ... • import always executes the entire file • Modules are still isolated environments • These variations are just manipulating names
File names have to follow the rules 24 # good.py ... • Comment: This mistake comes up a lot when teaching Python to newcomers • Must be a valid identifier name • Also: avoid non-ASCII characters # 2bad.py ... Yes No
It is standard practice for package and module names to be concise and lowercase 25 foo.py • Use a leading underscore for modules that are meant to be private or internal MyFooModule.py not _foo.py • Don't use names that match common standard library modules (confusing) projectname/ math.py
26 >>> import sys >>> sys.path ['', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/site-packages'] • Sometimes you might hack it import sys sys.path.append("/project/foo/myfiles") • If a file isn't on the path, it won't import ... although doing so feels "dirty"
• Modules only get loaded once >>> import spam >>> import sys >>> 'spam' in sys.modules True >>> sys.modules['spam'] <module 'spam' from 'spam.py'> >>> • There's a cache behind the scenes • Consequence: If you make a change to the source and repeat the import, nothing happens (often frustrating to newcomers)
• You can force-reload a module, but you're never supposed to do it >>> from importlib import reload >>> reload(spam) <module 'spam' from 'spam.py'> >>> • Apparently zombies are spawned if you do this • No, seriously. • Don't. Do. It.
If a file might run as a main program, do this 29 # spam.py ... if __name__ == '__main__': # Running as the main program ... • Such code won't run on library import import spam # Main code doesn't execute bash % python spam.py # Main code executes
larger collections of code, it is usually desirable to organize modules into a hierarchy spam/ foo.py bar/ grok.py ... 30 • To do it, you just add __init__.py files spam/ __init__.py foo.py bar/ __init__.py grok.py ...
• Import works the same way, multiple levels import spam.foo from spam.bar import grok 31 • The __init__.py files import at each level • Apparently you can do things in those files • We'll get to that
is "better?" • One .py file with 20 classes and 10000 lines? • 20 .py files, each containing a single class? • Most programmers prefer the latter • Smaller source files are easier to maintain 34
is better? • 20 files all defined at the top-level 35 foo.py bar.py grok.py • 20 files grouped in a directory spam/ foo.py bar.py grok.py • Clearly, latter option is easier to manage
is better? • One module import 36 • Importing dozens of submodules from spam import Foo, Bar, Grok • I prefer the former (although it depends) • "Fits my brain" from spam.foo import Foo from spam.bar import Bar from spam.grok import Grok
• Modules are easy--a single file • Packages are hard--multiple related files • Some Issues • Code organization • Connections between submodules • Desired usage • It can get messy 37
• Don't use implicit relative imports in packages spam/ __init__.py foo.py bar.py 38 • Example : # bar.py import foo # Relative import of foo submodule • It "works" in Python 2, but not Python 3
Alternative: Use an absolute module import spam/ __init__.py foo.py bar.py 39 • Example : # bar.py from spam import foo • Notice use of top-level package name • I don't really like it (verbose, fragile)
• A better approach spam/ __init__.py foo.py bar.py 40 • Example: # bar.py from . import foo # Import from same level • Leading dots (.) used to move up hierarchy from . import foo # Loads ./foo.py from .. import foo # Loads ../foo.py from ..grok import foo # Loads ../grok/foo.py
• Allow packages to be easily renamed spam/ __init__.py foo.py bar.py 41 • Explicit relative imports still work unchanged # bar.py from . import foo # Import from same level grok/ __init__.py foo.py bar.py • Useful for moving code around, versioning, etc.
PEP-8 predates explicit relative imports • I think its advice is sketchy on this topic • Please use explicit relative imports • They ARE used in the standard library
What are you supposed to do in those files? • Claim: I think they should mainly be used to stitch together multiple source files into a "unified" top-level import (if desired) • Example: Combining multiple Python files, building modules involving C extensions, etc.
Consider two submodules in a package 49 spam/ foo.py bar.py # foo.py class Foo(object): ... ... # bar.py class Bar(object): ... ... • Suppose you want to combine them
Combine in __init__.py 50 spam/ foo.py bar.py # foo.py class Foo(object): ... ... # bar.py class Bar(object): ... ... # __init__.py from .foo import Foo from .bar import Bar __init__.py
The collections "module" • It's actually a package with a few components 52 deque defaultdict _collections.so Container Hashable Mapping ... _collections_abc.py collections/__init__.py from _collections import ( deque, defaultdict ) from _collections_abc import * class OrdererDict(dict): ... class Counter(dict): ...
The last step is subtle 54 __all__ = (foo.__all__ + bar.__all__) • Ensures proper propagation of exported symbols to the top level of the package foo.py bar.py __all__ = ['Foo'] __all__ = ['Bar'] spam.py __all__ = ['Foo', 'Bar']
• I sometimes use an explicit export decorator 56 # spam/__init__.py __all__ = [] def export(defn): globals()[defn.__name__] = defn __all__.append(defn.__name__) return defn from . import foo from . import bar • Will use it to tag exported definitions • Might use it for more (depends)
• Example usage 57 # spam/foo.py from . import export @export def blah(): ... @export class Foo(object): ... • Benefit: exported symbols are clearly marked in the source code.
Should __init__.py import the universe? • For small libraries, who cares? • For large framework, maybe not (expensive) • Will return to this a bit later • For now: Think about about it 58
• Is this good style? 60 spam/ __init__.py # __init__.py class Foo(object): ... class Bar(object): ... • A one file package where everything is put inside __init__.py • It feels sort of "wrong" • __init__ connotes initialization, not implementation
Packages define an internal __path__ variable 61 >>> import xml >>> xml.__path__ ['/usr/local/lib/python3.4/xml'] >>> • It defines where submodules are located >>> import xml.etree >>> xml.etree.__file__ '/usr/local/lib/python3.4/xml/etree/__init__.py' >>> • Packages can hack it (in __init__.py) __path__.append('/some/additional/path')
A package can "upgrade" itself on import 62 # xml/__init__.py try: import _xmlplus import sys sys.modules[__name__] = _xmlplus except ImportError: pass • Idea: Replace the sys.modules entry with a "better" version of the package (if available) • FYI: xml package in Python2.7 does this
• Monkeypatching other modules on import? • Other initialization (logging, etc.) • My advice: Stay away. Far away. • Simple __init__.py == good __init__.py 63
python -m module • Runs a module as a main program 65 spam/ __init__.py foo.py bar.py bash % python3 -m spam.foo # Runs spam.foo as main • It's a bit special in that package relative imports and other features continue to work as usual
I like the -m option a lot • Makes the Python version explicit 67 bash % python3 -m pip install package bash % pip install package vs Rant: I can't count the number of times I've had to debug someone's Python installation because they're running some kind of "script", but they have no idea what Python it's actually attached to. The -m option avoids this.
__main__.py designates main for a package • Also makes a package directory executable 68 spam/ __init__.py __main__.py # Main program foo.py bar.py bash % python3 -m spam # Run package as main • Explicitly marks the entry point (good) • Useful for a variety of other purposes
Wrapper • Make a tool that wraps around a script • Examples: 70 bash % python3 -m profile someprogram.py bash % python3 -m pdb someprogram.py bash % python3 -m coverage run someprogram.py bash % python3 -m trace --trace someprogram.py ... • Many programming tools work this way
Variant, Python can execute a raw directory • Must contain __main__.py 72 spam/ foo.py bar.py __main__.py bash % python3 spam • This also applies to zip files bash % python3 -m zipfile -c spam.zip spam/* bash % python3 spam.zip
Obscure fact: you can prepend a zip file with #! to make it executable like a script (since Py2.6) 73 spam/ foo.py bar.py __main__.py bash % python3 -m zipfile -c spam.zip spam/* bash % echo -e '#!/usr/bin/env python3\n' > spamapp bash % cat spam.zip >>spamapp bash % chmod +x spamapp bash % ./spamapp • See PEP-441 for improved support of this
• Almost every tricky problem concerning modules/packages is related to sys.path 75 >>> import sys >>> sys.path ['', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload', '/usr/local/lib/python3.4/site-packages'] • Not on sys.path? Won't import. End of story. • Package managers/install tools love sys.path
• Python looks for many different kinds of files 79 >>> import spam • What it looks for (in each path directory) spam/ spam.cpython-34m.so spam.abi3.so spam.so spam.py __pycache__/spam.cpython-34.pyc spam.pyc • Run python3 -vv to see verbose output Package directory C Extensions (not allowed in .zip/.egg) Python source file Compiled Python
Path settings of a base Python installation 81 bash % python3 -S # -S skips site.py initialization >>> sys.path [ '', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4/', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload' ] >>> • These define the location of the standard library
Python binary location determines the prefix 83 bash % which python3 /usr/local/bin/python3 bash % sys.prefix = '/usr/local' • However, it's far more nuanced than this • Environment variable check • Search for "installation" landmarks • Virtual environments
sys.prefix is hard-coded into python (getpath.c) 87 /* getpath.c */ #ifndef PREFIX #define PREFIX "/usr/local" #endif #ifndef EXEC_PREFIX #define EXEC_PREFIX PREFIX #endif • This is set during compilation/configuration
of sys.prefix is a major part of tools that package Python in custom ways • Historically: virtualenv (Python 2) • Modern: pyvenv (Python 3, in standard library) • Of possible use in other settings (embedding, etc.) 88
Makes a Python virtual environment 92 bash % python3 -m venv spam spam/ pyvenv.cfg bin/ activate easy_install pip python3 include/ ... lib/ python3.4/ site-packages/ • A fresh "install" with no third-party packages • Includes python, pip, easy_install for setting up a new environment • I prefer 'python3 -m venv' over the script 'pyvenv'
Suppose you have a virtual environment 93 /Users/ beazley/ mypython/ pyvenv.cfg bin/ python3 lib/ python3.4/ ... site-packages/ • venv site-packages gets used instead of defaults
Variant: Include system site-packages 95 bash % python3 -m venv --system-site-packages mypython bash % mypython/bin/python3 >>> import sys >>> sys.path ['', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload', '/Users/beazley/mypython/lib/python3.4/site-packages', '/Users/beazley/.local/lib/python3.4/site-packages', '/usr/local/lib/python3.4/site-packages'] >>> Get the system site- packages and that of the virtual environment
A further technique of extending sys.path • Make a file with a list of additional directories 96 • Copy this file to any site-packages directory • All directories that exist are added to sys.path # foo.pth ./spam/grok ./blah/whatever
.pth files mainly used by package managers to install packages in additional directories • Example: adding '.egg' files to the path 98 >>> sys.path ['', '/usr/local/lib/python3.4/site-packages/ply-3.4-py3.4.egg', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload', ... ] • But, it gets even better!
• Example: setuptools.pth 99 import sys; sys.__plen = len(sys.path) ./ply-3.4-py3.4.egg import sys; new=sys.path[sys.__plen:]; del sys.path \ [sys.__plen:]; p=getattr(sys,'__egginsert',0); \ sys.path[p:p]=new; sys.__egginsert = p+len(new) • Any line starting with 'import' is executed • Package managers and extensions can use this to perform automagic steps upon Python startup • No patching of other files required
steps of site.py initialization • import sitecustomize • import usercustomize • ImportError silently ignored (if not present) • Both imports may further change sys.path 101
First path component is same directory as the running script (or current working directory) • It gets added last 102 bash % python3 programs/script.py >>> import sys >>> sys.path ['/Users/beazley/programs/', '/usr/local/lib/python34.zip', '/usr/local/lib/python3.4', '/usr/local/lib/python3.4/plat-darwin', '/usr/local/lib/python3.4/lib-dynload', ... ] Added last
easy_install, pip, conda, etc. • They all basically work within this environment • Installation into site-packages, etc. • Differences concern locating, downloading, building, dependencies, and other aspects. • Do I want to discuss further? Nope. 104
• Bah, you don't even need it! 106 spam/ foo.py bar.py • It all works fine without it! (No, Really) >>> import spam.foo >>> import spam.bar >>> spam.foo <module 'spam.foo' from 'spam/foo.py'> >>> • Wha!?!??? (Don't try in Python 2)
Omit __init__.py and you get a "namespace" 107 spam/ foo.py bar.py >>> import spam >>> spam <module 'grok' (namespace)> >>> • A namespace for what? • For building an extensible library of course!
Suppose you have two directories like this 108 spam_foo/ spam/ foo.py spam_bar/ spam/ bar.py • Both directories contain the same top-level package name, but different subparts same package defined in each directory
Put both directories on sys.path. 109 >>> import sys >>> sys.path.extend(['spam_foo','spam_bar']) >>> • Now, try some imports--watch the magic! >>> import spam.foo >>> import spam.bar >>> spam.foo <module 'spam.foo' from 'spam_foo/spam/foo.py'> >>> spam.bar <module 'spam.bar' from 'spam_bar/spam/bar.py'> >>> • Two directories become one!
• Packages have a magic __path__ variable 110 >>> import xml >>> xml.__path__ ['/usr/local/lib/python3.4/xml'] >>> • It's a list of directories searched for submodules • For a namespace, all matching paths get collected >>> spam.__path__ _NamespacePath(['spam_foo/spam', 'spam_bar/spam']) >>> • Only works if no __init__.py in top level
Nutshell 114 • There is a framework core telly/ __init__.py ... • There is a plugin area ("Tubbytronic Superdome") telly/ __init__.py ... tubbytronic/ laalaa.py ...
• Telly allows user-specified plugins (in $HOME) ~/.telly/ telly-dipsy/ tubbytronic/ dipsy.py telly-po/ tubbytronic/ po.py • Not installed as part of main package
• Figure out some way to unify all of the plugins in the same namespace >>> from telly.tubbytronic import laalaa >>> from telly.tubbytronic import dipsy >>> from telly.tubbytronic import po >>> • Even though the plugins are coming from separately installed directories
Just a bit of __path__ hacking # telly/__init__.py import os import os.path user_plugins = os.path.expanduser('~/.telly') if os.path.exists(user_plugins): plugins = os.listdir(user_plugins) for plugin in plugins: __path__.append(os.path.join(user_plugins, plugin)) • Does it work?
Namespace packages are kind of insane • Only thing more insane: Python 2 implementation of the same thing (involving setuptools, etc.) • One concern: Packages now "work" if users forget to include __init__.py files • Wonder if they know how much magic happens
Module? • A file of source code • A namespace • Container of global variables • Execution environment for statements • Most fundamental part of a program? 123
A module is an object (you can make one) >>> from types import ModuleType >>> spam = ModuleType('spam') >>> spam <module 'spam'> >>> 124 • It wraps around a dictionary >>> spam.__dict__ {'__loader__': None, '__doc__': None, '__name__': 'spam', '__spec__': None, '__package__': None} >>>
• A package is just a module with two defined (non-None) attributes 126 __package__ # Name of the package __path__ # Search path for subcomponents • Otherwise, it's the same object >>> import xml >>> xml.__package__ 'xml' >>> xml.__path__ ['/usr/local/lib/python3.4/xml'] >>> type(xml) <class 'module'> >>>
import creates a module object • Executes source code inside the module • Assigns the module object to a variable >>> import spam >>> spam <module 'spam' from 'spam.py'> >>> 127 • Creation is far more simple than you think
Modules are cached. This is checked first import sys, types def import_module(modname): if modname in sys.modules: return sys.modules[modname] ... mod = types.ModuleType(modname) mod.__file__ = sourcepath sys.modules[modname] = mod code = compile(sourcecode, sourcepath, 'exec') exec(code, mod.__dict__) return sys.modules[modname] 130 • New module put in cache prior to exec
The cache is a critical component of import • There are some tricky edge cases • Advanced import-related code might have to interact with it directly 131
• Definition/import order matters # foo.py import bar def spam(): ... 133 # bar.py import foo x = foo.spam() • Fail! >>> import foo Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/Users/beazley/.../foo.py", line 3, in <module> import bar File "/Users/beazley/.../bar.py", line 5, in <module> x = foo.spam() AttributeError: 'module' object has no attribute 'spam' >>>
• Definition/import order matters # foo.py import bar def spam(): ... 134 # bar.py import foo x = foo.spam() • Follow the control flow • A possible "fix" (move the import) # foo.py def spam(): ... import bar # bar.py import foo x = foo.spam() (Not Defined!) "ARG!!!!!" swap
Cycles • Cyclic imports in packages # spam/foo.py from . import bar ... 135 # spam/bar.py from . import foo ... • This crashes outright >>> import spam.foo Traceback (most recent call last): File "<stdin>", line 1, in <module> File "...spam/foo.py", line 1, in <module> from . import bar File "...spam/bar.py", line 1, in <module> from . import foo ImportError: cannot import name 'foo' >>>
Cycles • Problem: Reference to a submodule only get created after the entire submodule imports # spam/foo.py from . import bar ... 136 # spam/bar.py from . import foo ... spam.foo spam.bar spam package import tries to locate "spam.foo", but the symbol hasn't been created yet
Cycles • Can "fix" by realizing that sys.modules holds submodules as they are executing 137 # spam/bar.py try: from . import foo except ImportError: import sys foo = sys.modules[__package__ + '.foo'] • Commentary: This is a fairly obscure corner case--try to avoid import cycles if you can. That said, I have had to do this once in real-world production code.
• Imports can be buried in functions def evil(): import foo ... x = foo.spam() 138 • Functions can run in separate threads from threading import Thread t1 = Thread(target=evil) t2 = Thread(target=evil) t1.start() t2.start() • Concurrent imports? Yikes!
imports are locked 141 from threading import RLock _import_lock = RLock() def import_module(modname): with _import_lock: if modname in sys.modules: return sys.modules[modname] ... • Such a lock exists (for real) >>> import imp >>> imp.acquire_lock() >>> imp.release_lock() • Note: Not the same as the infamous GIL
Actual implementation is a bit more nuanced • Global import lock is only held briefly • Each module has its own dedicated lock • Threads can import different mods at same time • Deadlock detection (concurrent circular imports) • Advice: DON'T FREAKING DO THAT! 142
144 >>> spam = __import__('spam') >>> spam <module 'spam' from 'spam.py'> >>> • A better alternative: importlib.import_module() # Same as: import spam spam = importlib.import_module('spam') # Same as: from . import spam spam = importlib.import_module('.spam', __package__) • Direct use is possible, but discouraged
points: • Modules are objects • Basically just a dictionary (globals) • Importing is just exec() in disguise • Variations on import play with names • Tricky corner cases (threads, cycles, etc.) 146 • Modules are fundamentally simple
class Foo(object): ... # spam/bar.py class Bar(object): ... # spam/__init__.py from .foo import * from .bar import * Module Assembly (Reprise) • Consider: A package that stitches things together • It imports everything (might be slow)
spam >>> f = spam.Foo() Loaded Foo >>> f <spam.foo.Foo object at 0x100656f60> >>> Thought • What if subcomponents only load on demand? • No extra imports needed • Autoload happens behind the scenes
# List the exported symbols by module _submodule_exports = { '.foo' : ['Foo'], '.bar' : ['Bar'] } # Make a {name: modname } mapping _submodule_by_name = { name: modulename for modulename in _submodule_exports for name in _submodule_exports[modulename] } Lazy Module Assembly • Alternative approach • This is not actually importing anything...
# List the exported symbols by module _submodule_exports = { '.foo' : ['Foo'], '.bar' : ['Bar'] } # Make a {name: modname } mapping _submodule_by_name = { name: modulename for modulename in _submodule_exports for name in _submodule_exports[modulename] } Lazy Module Assembly • Alternative approach • It builds symbol-module name map { 'Foo' : '.foo', 'Bar': '.bar' ... }
spam >>> f = spam.Foo() Loaded Foo >>> f <spam.foo.Foo object at 0x100656f60> >>> from spam import Bar Loaded Bar >>> Bar <class 'spam.bar.Bar'> >>> Example • That's crazy! • Not my idea: Armin Ronacher • Werkzeug (http://werkzeug.pocoo.org)
An existing module can be reloaded 155 >>> import spam >>> from importlib import reload >>> reload(spam) <module 'spam' from 'spam.py'> >>> • As previously noted: zombies are spawned • Why?
Reloading in a nutshell 156 >>> import spam >>> code = open(spam.__file__, 'rb').read() >>> exec(code, spam.__dict__) >>> • It simply re-executes the source code in the already existing module dictionary • It doesn't even bother to clean up the dict • So, what can go wrong?
• Suppose you have a package 158 # spam/__init__.py print('Loading spam') from . import foo from . import bar • What happens to the submodules on reload? >>> import spam Loading spam >>> importlib.reload(spam) Loading spam <module 'spam' from 'spam/__init__.py'> >>> • Nothing happens: They aren't reloaded
• Suppose you have a class 159 # spam.py class Spam(object): def yow(self): print('Yow!') import spam a = spam.Spam() • Now, you change it and reload # spam.py class Spam(object): def yow(self): print('Moar Yow!') reload(spam) b = spam.Spam() a.yow() # ???? b.yow() # ????
• Suppose you have a class 160 # spam.py class Spam(object): def yow(self): print('Yow!') import spam a = spam.Spam() • Now, you change it and reload # spam.py class Spam(object): def yow(self): print('Moar Yow!') reload(spam) b = spam.Spam() a.yow() # Yow! b.yow() # Moar Yow!
• Existing instances keep their original class 161 class Spam(object): def yow(self): print('Yow!') b.__class__ • New instances will use the new class class Spam(object): def yow(self): print('Moar Yow!') >>> a.yow() Yow! >>> b.yow() Moar Yow! >>> type(a) == type(b) False >>> a.__class__
You might have multiple implementations of the code actively in use at the same time • Maybe it doesn't matter • Maybe it causes your head to explode • No, spawned zombies eat your brain 162
Modules can detect/prevent reloading 163 # spam.py if 'foo' in globals(): raise ImportError('reload not allowed') def foo(): ... • Idea: Look for names already defined in globals() • Recall: module dict is not cleared on reload
Packages could reload their subcomponents 164 # spam/__init__.py if 'foo' in globals(): from importlib import reload foo = reload(foo) bar = reload(bar) else: from . import foo from . import bar • Ugh. No. Please don't.
• You might try to make it work with hacks 165 import weakref class Spam(object): if 'Spam' in globals(): _instances = Spam._instances else: _instances = weakref.WeakSet() def __init__(self): Spam._instances.add(self) def yow(self): print('Yow!') for instance in Spam._instances: instance.__class__ = Spam • Will make "code review" more stimulating
safe/sane way to reload is to restart • Your time is probably better spent trying to devise a sane shutdown/restart process to bring in code changes • Possibly managed by some kind of supervisor process or other mechanism 167
follows has been an actively changing part of Python • It assumes Python 3.5 or newer • It might be changed again • Primary goal: Peek behind the covers a little bit 169
sys.path is the most visible configuration of the module/package system to users 170 >>> import sys >>> sys.path ['', '/usr/local/lib/python35.zip', '/usr/local/lib/python3.5', '/usr/local/lib/python3.5/plat-darwin', '/usr/local/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/site-packages'] • It is not the complete picture • In fact, it is a small part of the bigger picture
is actually controlled by sys.meta_path 171 >>> import sys >>> sys.meta_path [<class '_frozen_importlib.BuiltinImporter'>, <class '_frozen_importlib.FrozenImporter'>, <class '_frozen_importlib.PathFinder'>] >>> • It's a list of "importers" • When you import, they are consulted in order
A module spec can be useful all by itself • Consider: (Inspired by Armin Ronacher [1]) 175 # spam.py try: import foo except ImportError: import simplefoo as foo # foo.py import bar # Not found Scenario: Code that tests to see if a module can be imported. If not, it falls back to an alternative. [1] http://lucumr.pocoo.org/2011/9/21/python-import-blackbox/
A module spec can be useful all by itself • A Reformulation 177 # spam.py from importlib.util import find_spec if find_spec('foo'): import foo else: import simplefoo • If the module can be found, it will import • A "look before you leap" for imports
Example: 178 >>> import spam Traceback (most recent call last): File "<stdin>", line 1, in <module> File ".../spam.py", line 3, in <module> import foo File ".../foo.py", line 1, in <module> import bar ImportError: No module named 'bar' >>> • It's a much better error • Directly points at the problem
A separate "loader" object lets you do more 179 >>> spec = find_spec('socket') >>> spec.loader <_frozen_importlib.SourceFileLoader object at 0x1007706a0> >>> • Example: Pull the source code >>> src = spec.loader.get_source(spec.name) >>> • More importantly: loaders actually create the imported module
Example of creation 180 module = spec.loader.create_module(spec) if not module: module = types.ModuleType(spec.name) module.__file__ = spec.origin module.__loader__ = spec.loader module.__package__ = spec.parent module.__path__ = spec.submodule_search_locations module.__spec__ = spec • But don't do that... it's already in the library (py3.5) # Create the module from importlib.util import module_from_spec module = module_from_spec(spec)
• Module creation currently has a split personality • Legacy Interface: Python 3.3 and earlier 181 module = loader.load_module() • Modern Interface: Python 3.4 and newer module = loader.create_module(spec) if not module: # You're on your own. Make a module object # however you want ... sys.modules[spec.name] = module loader.exec_module(module) • Legacy interface still used for all non-Python modules (builtins, C extensions, etc.)
module loading technique is better • Decouples module creation/execution • Allows for more powerful programming techniques involving modules • Far fewer "hacks" • Let's see an example 183
>>> # Import the module >>> socket = lazy_import('socket') >>> socket <module 'socket' from '/usr/local/lib/python3.5/socket.py'> >>> dir(socket) ['__doc__', '__loader__', '__name__', '__package__', '__spec__'] • Idea: create a module that doesn't execute itself until it is actually used for the first time >>> socket.AF_INET <AddressFamily.AF_INET: 2> >>> dir(socket) ['AF_APPLETALK', 'AF_DECnet', 'AF_INET', 'AF_INET6', ... ] >>> • Now, access it
import importlib.util, sys def lazy_import(name): # If already loaded, return the module if name in sys.modules: return sys.modules[name] # Not loaded. Find the spec spec = importlib.util.find_spec(name) if not spec: raise ImportError('No module %r' % name) # Check for compatibility if not hasattr(spec.loader, 'exec_module'): raise ImportError('Not supported') module = sys.modules[name] = _LazyModule(spec) return module • A utility function to make the "import"
• Actually a somewhat old (and new) idea • Goal is to reduce startup time • Python 2 implementation (Phillip Eby) • https://pypi.python.org/pypi/Importing • Significantly more "hacky" (involves reload) • There's a LazyLoader coming in Python 3.5
• As noted: Python tries to find a "spec" # importlib.util def find_spec(modname): for imp in sys.meta_path: spec = imp.find_spec(modname) if spec: return spec return None • You can also plug into this machinery to do interesting things as well
Idle thought: Wouldn't it be cool if unresolved imports would just automatically download from PyPI? >>> import requests Traceback (most recent call last): File "<stdin>", line 1, in <module> ImportError: No module named 'requests' >>> import autoinstall >>> import requests Installing requests >>> requests <module 'requests' from '...python3.4/site-packages/requests/ __init__.py'> >>> • Disclaimer: This is a HORRIBLE idea
• Thought: Could modules be imported from Redis? • Redis in a nutshell: a key/value server >>> import redis >>> r = redis.Redis() >>> r.set('bar', 'hello') True >>> r.get('bar') b'hello' >>> • Challenge: load code from it?
Yes, yes, sys.path. 200 >>> import sys >>> sys.path ['', '/usr/local/lib/python35.zip', '/usr/local/lib/python3.5', '/usr/local/lib/python3.5/plat-darwin', '/usr/local/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/site-packages'] • There is yet another piece of the puzzle
entry on sys.path is tested against a list of "path hook" functions 201 >>> import sys >>> sys.path_hooks [ <class 'zipimport.zipimporter'>, <function FileFinder..path_hook_for_FileFinder at 0x1003afbf8> ] >>> • Functions merely decide whether or not they can handle a particular path
202 >>> path = '/usr/local/lib/python3.5' >>> finder = sys.path_hooks[0](path) Traceback (most recent call last): File "<stdin>", line 1, in <module> zipimport.ZipImportError: not a Zip file >>> finder = sys.path_hooks[1](path) >>> finder FileFinder('/usr/local/lib/python3.5') >>> • Idea: Python uses the path_hooks to associate a module finder with each path entry
Path finders are used to locate modules 203 >>> finder FileFinder('/usr/local/lib/python3.5') >>> finder.find_spec('datetime') ModuleSpec(name='datetime', loader=<_frozen_importlib.SourceFileLoader object at 0x10068b7f0>, origin='/usr/local/lib/python3.5/datetime.py') >>> • Uses the same machinery as before (ModuleSpec)
What happens during import (roughly) 205 modname = 'somemodulename' for entry in sys.path: finder = sys.path_importer_cache[entry] if finder: spec = finder.find_spec(modname) if spec: break else: raise ImportError('No such module') ... # Load module from the spec ...
Naturally, you can hook into the sys.path machinery with your own custom code • Requires three components • A path hook • A finder • A loader • Example follows 206
• Example: Consider some Python code 207 spam/ foo.py bar.py • Make it available via a web server bash % cd spam bash % python3 -m http.server Serving HTTP on 0.0.0.0 port 8000 ... • Allow imports via sys.path import sys sys.path.append('http://someserver:8000') import foo
• Step 1: Write a hook to recognize URL paths 208 import re, urllib.request def url_hook(name): if not name.startswith(('http:', 'https:')): raise ImportError() data = urllib.request.urlopen(name).read().decode('utf-8') filenames = re.findall('[a-zA-Z_][a-zA-Z0-9_]*\.py', data) modnames = { name[:-3] for name in filenames } return UrlFinder(name, modnames) import sys sys.path_hooks.append(url_hook) • This makes an initial URL request, collects the names of all .py files it can find, creates a finder.
use: 211 >>> import sys >>> sys.path.append('http://localhost:8000') >>> import foo >>> foo <module 'foo' (http://localhost:8000/foo.py)> >>> • Bottom line: You can make custom paths • Not shown: Making this work with packages
There are a lot of moving parts • A good policy: Keep it as simple as possible • It's good to understand what's possible • In case you have to debug it
https://docs.python.org/3/reference/import.html • https://docs.python.org/3/library/importlib • Relevant PEPs PEP 273 - Import modules from zip archives PEP 302 - New import hooks PEP 338 - Executing modules as scripts PEP 366 - Main module explicit relative imports PEP 405 - Python virtual environments PEP 420 - Namespace packages PEP 441 - Improving Python ZIP application support PEP 451 - A ModuleSpec type for the import system
I hope you got some new ideas • Please feel free to contact me http://www.dabeaz.com • Also, I teach Python classes @dabeaz (Twitter) • Special Thanks: http://www.dabeaz.com/chicago A. Chourasia, Y. Tymciurak, P. Smith, E. Meschke, E. Zimmerman, JP Bader
dabeaz
sutetotanuki
hypermkt
ivargrimstad
waka292
hatsu38
usagimaru
nrslib
hal_spidernight
yasei_no_otoko
yusukebe
tsoshiro
vanstee
jonyablonski
eitanlees
rocio
reverentgeek
hatefulcrawdad
paulrobertlloyd
bkeepers
morganepeng
ammeep
sugarenia
danielanewman
