Python Packaging

Summarized talk about history of Python packaging ecosystem and the future of it.

1. Python Packaging Hong Minhee
2. Speaker: Hong Minhee • Wand, sphinxcontrib-httpdomain, libsass, Sider, Okydoky • Web developer • Python programmer since 2005 • CDNetworks, StyleShare, Crosspop • http://dahlia.kr/ • irc://irc.ozinger.org/hongminhee
4. Basic concepts
5. sys.path $ echo 'bar = "ok"' > /tmp/foo.py $ python Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path.append('/tmp') >>> import foo >>> foo.bar 'ok'
6. site-packages (dist-packages in Debian/Ubuntu) Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import sys >>> sys.path ['', '/usr/lib/python2.7', ..., '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/PIL'...]
7. site-packages (dist-packages in Debian/Ubuntu) $ ls -p /usr/local/lib/python2.7/dist-packages/ easy-install.pth Sphinx-1.1.3-py2.7.egg rsa/ rsa-3.0.1.egg-info tox-1.4.2-py2.7.egg chardet/ virtualenv-1.7.2-py2.7.egg chardet-1.0.1.egg-info docutils-0.9.1-py2.7.egg
8. *.pth $ cat /.../site-packages/easy-install.pth import sys; sys.__plen = len(sys.path) ./Sphinx-1.1.3-py2.7.egg ./docutils-0.9.1-py2.7.egg ./tox-1.4.2-py2.7.egg ./virtualenv-1.7.2-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys. path[sys.__plen:]; p=getattr(sys,'__egginsert', 0); sys.path[p:p]=new; sys.__egginsert = p+len (new)
9. History of Python packaging
10. The way the most of us have packaged $ cat requirements.txt Flask==0.9 SQLAlchemy==0.8.0 Wand==0.2.3 $ pip install -r requirements.txt
11. It is not packaging • No versioning • You can’t upload this on PyPI (Others cannot install it using pip or easy_install) • Has no package name • Doesn’t work at all if it contains any C extensions ◦ How is it built? ◦ No binary distributions • No metadata (Who made it? What does it for me?) • requirements.txt is not packaging but using packages
12. The distutils–packaging dynasty • distutils • setuptools (easy_install) • Distribute • pip • Distutils2 → packaging
13. The distutils–packaging dynasty • distutils: zip, tgz, tar.bz2 ({s,b}dist) • Platform specific: rpm, deb, msi • setuptools: egg (bdist) • pip: pybundle (sdist), wheel (bdist)
14. In the beginning, there was distutils $ tree . ├── foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 2 directories, 6 files
15. In the beginning, there was distutils from distutils.core import setup setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='minhee@dahlia.kr', url='http://example.com/', packages=['foo', 'foo.bar'])
16. In the beginning, there was distutils $ python setup.py sdist running sdist running check writing manifest file 'MANIFEST' creating Foo-1.0 ... making hard links in Foo-1.0... hard linking setup.py -> Foo-1.0 hard linking foo/__init__.py -> Foo-1.0/foo ... creating dist Creating tar archive removing 'Foo-1.0' (and everything under it)
17. In the beginning, there was distutils $ tree . ├── MANIFEST ├── dist │ └── Foo-1.0.tar.gz ├── foo │ ├── __init__.py │ ├── bar │ │ ├── __init__.py │ │ └── mod.py │ ├── exc.py │ └── util.py └── setup.py 3 directories, 8 files
18. In the beginning, there was distutils $ tar xvfz Foo-1.0.tar.gz ... $ cd Foo-1.0 $ python setup.py install running install running build ... running install_lib creating /.../python2.7/site-packages/foo copying build/lib/foo/__init__.py -> /.../python2.7/site- packages/foo creating /.../python2.7/site-packages/foo/bar ... copying build/lib/foo/exc.py -> /.../python2.7/site-packages/foo copying build/lib/foo/util.py -> /.../python2.7/site-packages/foo
19. In the beginning, there was distutils $ tree lib/python2.7/site-packages/ lib/python2.7/site-packages/ └── foo ├── __init__.py ├── bar │ ├── __init__.py │ └── mod.py ├── exc.py └── util.py 2 directories, 5 files
20. In the beginning, there was distutils Python 2.7.2+ (default, Jul 20 2012...) [GCC 4.6.1] on linux2 Type "help", "copyright", "credits" or ... >>> import foo.bar.mod >>> foo.bar.mod.__doc__ 'Useful modules.'
21. setuptools: De facto standard • Built on top of distutils • easy_install • Dependency resolution • .egg binary distribution • Resource management • Dynamic discovery of plugins
22. setuptools: Built on top of distutils from distutils.core import setup setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='minhee@dahlia.kr', url='http://example.com/', packages=['foo', 'foo.bar'])
23. setuptools: Built on top of distutils from setuptools import setup, find_packages setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='minhee@dahlia.kr', url='http://example.com/', packages=find_packages(), install_requires=['Flask', 'Wand>=0. 2.1'])
24. setuptools: Dependency resolution $ python setup.py install running install running bdist_egg running egg_info creating Foo.egg-info writing requirements to Foo.egg-info/requires.txt writing Foo.egg-info/PKG-INFO writing top-level names to Foo.egg-info/top_level.txt writing dependency_links to Foo.egg-info/dependency_links. txt writing manifest file 'Foo.egg-info/SOURCES.txt' ...
25. setuptools: Dependency resolution Installed /.../python2.7/site-packages/Foo-1.0-py2.7.egg Processing dependencies for Foo==1.0 Searching for Wand>=0.2.1 Reading https://pypi.crate.io/simple/Wand/ Best match: Wand 0.2.3 Downloading ... Processing Wand-0.2.3.tar.gz ... Adding Wand 0.2.3 to easy-install.pth file Installed /.../python2.7/site-packages/Wand-0.2.3-py2.7. egg
26. setuptools: Dependency resolution $ ls -p lib/python2.7/site-packages/ Flask-0.9-py2.7.egg/ easy-install.pth Foo-1.0-py2.7.egg pip-1.2.1-py2.7.egg/ Jinja2-2.6-py2.7.egg/ setuptools-0.6c11-py2.7.egg Wand-0.2.3-py2.7.egg setuptools.pth Werkzeug-0.8.3-py2.7.egg/
27. setuptools: Dependency resolution $ cat lib/python2.7/site-packages/easy-install.pth import sys; sys.__plen = len(sys.path) ./setuptools-0.6c11-py2.7.egg ./pip-1.2.1-py2.7.egg ./Foo-1.0-py2.7.egg ./Wand-0.2.3-py2.7.egg ./Flask-0.9-py2.7.egg ./Jinja2-2.6-py2.7.egg ./Werkzeug-0.8.3-py2.7.egg import sys; new=sys.path[sys.__plen:]; del sys.path[sys. __plen:]; p=getattr(sys,'__egginsert',0); sys.path[p:p] =new; sys.__egginsert = p+len(new)
28. setuptools: Submit to PyPI $ python setup.py register ... $ python setup.py sdist upload ...
29. $ easy_install Foo ... $ easy_install "Foo >= 1.0" ... $ easy_install "Foo == 1.0" ... easy_install does install
30. C:\> easy_install libsass Searching for libsass Reading http://pypi.python.org/simple/libsass/ Best match: libsass 0.2.4 Downloading http://.../libsass-0.2.4-py2.7-win-amd64.egg Processing libsass-0.2.4-py2.7-win-amd64.egg Moving libsass-0.2.4-py2.7-win-amd64.egg to C:\...\site-packages Adding libsass 0.2.4 to easy-install.pth file Installing sassc.py script to C:\...\Scripts Deleting C:\...\Scripts\sassc.py Installing sassc-script.py script to C:\...\Scripts Installing sassc.exe script to C:\...\Scripts Installed C:\...\site-packages\libsass-0.2.4-py2.7-win-amd64.egg Processing dependencies for libsass Finished processing dependencies for libsass egg: Binary distribution
31. pip: Successor of easy_install • Replaces easy_install (not setuptools) • Can uninstall packages • Install packages w/ atomic transaction ◦ Install everything ◦ or nothing • Can freeze current site-packages ◦ Widely used pattern to generate requirements. txt • Very useful install -r option • No easy-install.pth or such
32. $ easy_install Flask Searching for Flask Reading https://pypi.crate.io/simple/Flask/ Best match: Flask 0.9 Downloading ... Processing Flask-0.9.tar.gz ... Adding Flask 0.9 to easy-install.pth file Installed /.../lib/python2.7/site-packages/Flask-0.9-py2.7.egg Processing dependencies for Flask Searching for Jinja2>=2.4 Reading https://pypi.crate.io/simple/Jinja2/ Best match: Jinja2 2.6 Downloading ... ^Cinterrupted pip: Atomic installation
33. $ ls lib/python2.7/site-packages/ Flask-0.9-py2.7.egg pip- 1.2.1-py2.7.egg distribute-0.6.28-py2.7.egg setuptools.pth easy-install.pth pip: Atomic installation
34. pip: Atomic installation $ pip install Flask Downloading/unpacking Flask Downloading Flask-0.9.tar.gz (481kB): 481kB downloaded Running setup.py egg_info for package Flask Downloading/unpacking Werkzeug>=0.7 (from Flask) Downloading Werkzeug-0.8.3.tar.gz (1.1MB): 1.1MB downloaded Running setup.py egg_info for package Werkzeug Downloading/unpacking Jinja2>=2.4 (from Flask) Downloading Jinja2-2.6.tar.gz (389kB): 208kB downloaded Operation cancelled by user Storing complete log in /Users/dahlia/.pip/pip.log
35. pip: Atomic installation $ ls lib/python2.7/site-packages/ distribute-0.6.28-py2.7.egg pip- 1.2.1-py2.7.egg easy-install.pth setuptools.pth
36. pip: Downsides? • Can’t install egg packages • Hard to use pip on Windows • Or you need entire C build toolchain to install ordinary Python packages at least ◦ Visual Studio ◦ cygwin? mingw? • Discourage to declare package metadata using setup.py
37. Distribute: Maintained fork of setuptools • setuptools hasn’t been maintained since 2006 • setuptools doesn’t provide recent Python versions • Distribute is a fork of setuptools • Hence it provides the same namespace to setuptools >>> import setuptools >>> setuptools <module 'setuptools' from /.../site-packages/distribute- 0.6.30-py2.7.egg/setuptools/__init__.pyc'>
38. Future of Python packaging
39. packaging • Also known as Distutils2 • To be merged to Python standard library • Declarative packaging via setup.cfg • Eliminate imperative packaging (setup.py) • You can’t use it for your package right now • Though it would be possible soon :-)
40. setup.py can do everything! • Ordinary Python script • That means it’s Turing-complete • It gives powerful flexibility ◦ Sync long_description field with README file ◦ Conditional dependencies using sys.platform and if keyword ◦ Platform-aware building process by extending distutils
41. setup.py can do everything! import sys from setuptools import setup, find_packages install_requires = ['Flask', 'Wand>=0.2.1'] if sys.platform == 'win32': install_requires.append('pywin32') setup(name='Foo', version='1.0', description='Very useful thing', author='Hong Minhee', author_email='minhee@dahlia.kr', url='http://example.com/', packages=find_packages(), install_requires=install_requires)
42. setup.py knows nothing… • You can even implement a PHP-to-Python compiler in the script and generate Python files just-in-time • Nothing can be assumed with setup.py • Cannot be indexed • You need to execute the script to get metadata • Exactly, one of possible metadata variants • The root cause of why installing Python packages is so slow
43. setup.py knows nothing… from random import randrange from distutils.core import setup random_value = str(randrange(1000)) module_name = 'random_module_name_' + random_value with open(module_name + '.py', 'w') as pyfile: print >> pyfile, 'value = ' + random_value setup(name='Crazy', description='You cannot index this', py_modules=[module_name])
44. setup.py knows nothing… $ python setup.py sdist running sdist running check writing manifest file 'MANIFEST' creating Crazy-0.0.0 making hard links in Crazy-0.0.0... hard linking random_module_name_832.py -> Crazy-0.0.0 hard linking setup.py -> Crazy-0.0.0 creating dist Creating tar archive removing 'Crazy-0.0.0' (and everything under it)
45. Environment markers (PEP 345) Requires-Dist: pywin32 (>1.0); sys.platform == 'win32' Obsoletes-Dist: pywin31; sys.platform == 'win32' Requires-Dist: foo (1,!=1.3); platform.machine == 'i386' Requires-Dist: bar; python_version == '2.4' or python_version == '2.5' Requires-External: libxslt; 'linux' in sys.platform
46. TL;DR: Package management • Install distribute (instead of setuptools) first • easy_install pip virtualenv • Use pip to install and uninstall • Use easy_install to install C extensions • Isolate site-packages per project using virtualenv
47. TL;DR: Distributing packages • Write setup.py script • Support both distribute and setuptools (for Python 2 packages) • Support distribute (for Python 3 packages) • Submit to PyPI using python setup.py register && python setup.py sdist upload command • Provide binary egg packages for C extensions
48. TL;DR: The future of Python ecosystem • Ready for packaging a.k.a. Distutils2 • Test your package on various Python VMs/versions (Python 2 and 3, CPython, PyPy…) • tox will help • Prefer pip over easy_install • Don’t make your setup.py dynamic • It will be your future headache
49. j.mp/pykr2013-packaging
50. Any questions?