Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Build and test wheel packages on Linux, OSX and...

Build and test wheel packages on Linux, OSX and Windows

Presentation given at PyCon 2015

Olivier Grisel

April 11, 2015
Tweet

More Decks by Olivier Grisel

Other Decks in Technology

Transcript

  1. Outline • The Wheel format • Testing under Linux and

    OSX with Travis-CI • Building OSX wheels • Testing under Windows with AppVeyor-CI • Building Windows wheels • Automating a release to PyPI • Tricky stuff: compilers on Windows and shared libraries
  2. What? • A standard distribution format for Python packages: https://www.python.org/dev/peps/pep-0427

    • Simple: ZIP archive with Python files + static metadata + compiled extensions (optional) • Can be installed by pip 1.4+ (current is 6.1) • Can be generated by your usual setup.py or pip
  3. Why? • Smaller packages: faster to download, even for pure-Python

    packages • Faster to install, even for pure-Python packages • Possible to install platform specific Python packages without a compiler nor header files for third party libraries
  4. $ time pip install numpy scipy scikit-learn >/dev/null real 0m9.034s

    user 0m6.355s sys 0m2.119s $ python -c "import sklearn; print(sklearn.__version__)" 0.16.0
  5. $ time pip install nose-1.3.6.tar.gz Processing ./nose-1.3.6.tar.gz Installing collected packages:

    nose Running setup.py install for nose Successfully installed nose-1.3.6 real 0m8.818s user 0m8.152s sys 0m0.494s
  6. $ time pip install nose-1.3.6-py3-none-any.whl Unpacking ./nose-1.3.6-py3-none-any.whl Installing collected packages:

    nose Successfully installed nose Cleaning up... real 0m0.568s user 0m0.447s sys 0m0.106s
  7. $ unzip nose-1.3.6-py3-none-any.whl Archive: nose-1.3.6-py3-none-any.whl inflating: nose/__init__.py inflating: nose/__main__.py inflating:

    nose/case.py inflating: nose/commands.py [...] inflating: nose-1.3.6.data/data/man/man1/nosetests.1 inflating: nose-1.3.6.dist-info/DESCRIPTION.rst inflating: nose-1.3.6.dist-info/entry_points.txt inflating: nose-1.3.6.dist-info/metadata.json inflating: nose-1.3.6.dist-info/top_level.txt inflating: nose-1.3.6.dist-info/WHEEL inflating: nose-1.3.6.dist-info/METADATA inflating: nose-1.3.6.dist-info/RECORD
  8. # In your setup.cfg: [bdist_wheel] universal = 1 # Then

    in a console: $ pip install wheel twine $ python setup.py bdist_wheel # At release time: $ python setup.py sdist bdist_wheel $ twine upload dist/* Universal Wheels Python 2 & 3, no compiled extensions
  9. .travis.yml to test on Linux language: python python: - "2.6"

    - "2.7" - "3.3" - "3.4" - "nightly" install: - "pip install ." script: - "(cd /tmp && nosetests -v mypackage)"
  10. .travis.yml to build and test OSX wheels language: objective-c env:

    matrix: - VERSION=2.7.9 - VERSION=3.3.5 - VERSION=3.4.3 install: - "source terryfy/travis_tools.sh" - "get_python_environment macpython $VERSION venv" - "pip install nose wheel" - "python setup.py bdist_wheel" - "pip install dist/*.whl" script: - "(cd /tmp && nosetests -v mypackage)"
  11. OSX and Linux on travis • Multi-OS support: • Use

    $TRAVIS_OS_NAME to make scripts OS aware • Still need to setup custom Python env under OSX: https://github.com/MacPython/terryfy os: - linux - osx
  12. environment: matrix: - PYTHON: "C:\\Python27" - PYTHON: "C:\\Python34" install: -

    "powershell ./install_python_and_pip.ps1” - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel” - "python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (without compiled extensions)
  13. environment: global: COMPILER_ENV: "cmd /E:ON /V:ON /C run_with_compilers.cmd" matrix: -

    PYTHON: "C:\\Python27" PYTHON_ARCH: "2.7.9" PYTHON_ARCH: "32" install: - "powershell ./install_python_and_pip.ps1" - "SET PATH=%PYTHON%;%PYTHON%\\Scripts;%PATH%" - "pip install nose wheel" - "%COMPILER_ENV% python setup.py bdist_wheel" - "pip install dist/*" build: false test_script: - "cd C:\\Temp" - "nosetests -v mypackage" appveyor.yml (with compiled extensions)
  14. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels
  15. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels export WHEELHOUSE_UPLOADER_USERNAME=mycloudaccountid export WHEELHOUSE_UPLOADER_SECRET=xxx python -m wheelhouse_uploader upload container_id
  16. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 python setup.py fetch_artifacts
  17. setup.cfg [wheelhouse_uploader] artifact_indexes= # OSX wheels built by travis (only

    for specific tags): # https://github.com/MacPython/scikit-learn-wheels http://wheels.scipy.org/ # Windows wheels buit by: # https://ci.appveyor.com/project/sklearn-ci/scikit-learn/ http://windows-wheels.scikit-learn.org/
  18. travis-ci OSX Rackspace CloudFiles wheels container for OS X Release

    Manager’s laptop appveyor-ci Windows Rackspace CloudFiles wheels container for Windows PyPI archive for official released wheels git checkout 0.16.0 rm -rf dist && python setup.py sdist fetch_artifacts twine upload dist/*
  19. Installing compilers on Windows Install MSVC Express 2008 / 2010

    + Windows SDK • Many non-scriptable manual steps • Windows SDK are several GBs in total to download http://scikit-learn.org/dev/install.html#building-on- windows
  20. MSVC for Python 2.7 [+] Small .msi file with a

    fixed download URL: http://download.microsoft.com/download/ 7/9/6/796EF2E4-801B-4FC4-AB28-B59FBF6D907B/ VCForPython27.msi [+] Support for 32 and 64 bit Python by default [-] Only C & C++, no fortran compiler (scipy…) [-] No support for Python 3.3 and 3.4 (need MSVC 2010) [+] Python 3.5 might be built with MSVC 2015 (forward compat)
  21. Carl Kleffner’s mingwpy • Static MinGW-w64 toolchain for 32 and

    64 bit • Links against MSVC runtime libs for binary compat with official Python interpreters from python.org • Provides gcc, g++ and gfortran • Still experimental but already used by WinPython, a wheel-based binary distribution • http://github.com/mingwpy (new, created this morning)
 https://bitbucket.org/carlkl/mingw-w64-for-python (old)
  22. External libraries • Examples: • lxml: libxml2 / libxslt •

    numpy & scipy: ATLAS, OpenBLAS or MKL • Under Windows: • Python ext: .pyd, Shared lib: .dll • Under OSX: • Python ext: .so, Shared lib: .dylib
  23. Possible solutions • Assume the library is present on the

    system: • BLAS / LAPACK for numpy & scipy under OSX • Use static linking: big .whl files • numpy with MKL by Christoph Gohlke • lxml under Windows (and OSX) • Embed the shared lib in the wheel archive
  24. Embedding a shared lib under OSX • delocate by Matthew

    Brett • https://github.com/matthew-brett/delocate
  25. $ delocate-wheel -v scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Fixing: scipy-0.14.0-cp34-cp34m-macosx_10_6_intel.whl Copied to package .dylibs

    directory: /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgcc_s.1.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libgfortran.3.dylib /usr/local/Cellar/gfortran/4.8.2/gfortran/lib/libquadmath.0.dylib
  26. Embedding a shared lib under Windows • Put the .dll

    files next to the .pyd files under the site-packages folder • Put the .dll files in the Scripts folder • Improve Python to have an official way to deploy .dll files shipped with wheels.