Running a Python Package Index for Raspberry Pi

Talk given at PackagingCon 2021

Transcript

1. @ben_nuttall Running a Python Package Index for Raspberry Pi
2. @ben_nuttall Ben Nuttall • Senior Software Engineer, BBC News Labs • Former Community Manager at Raspberry Pi • Based in Cambridgeshire, UK • bennuttall.com • twitter.com/ben_nuttall • github.com/bennuttall
3. @ben_nuttall Pure Python package Binary distribution (bdist) .whl Source distribution (sdist) .zip .tar.gz
4. @ben_nuttall Compiled Python package Python version 3.5 3.6 3.7 3.8 3.9 Windows (win32) Windows (amd64) Mac OS (x86_64) Linux (i686) Linux (x86_64) Source (sdist)
5. @ben_nuttall pip
6. @ben_nuttall pip
7. @ben_nuttall What is piwheels? • Open source project maintained by: – Ben Nuttall – Dave Jones • Tooling to automate building wheels of everything on PyPI – Wheels are pre-compiled binary distributions of Python packages • Pip-compatible index (PEP-503) – Can be used by pip as an "extra index" alongside PyPI • Tailored towards the Arm platform – linux_armv6l and linux_armv7l platform tags
8. @ben_nuttall piwheels Stretch (Python 3.5) Buster (Python 3.7) Bullseye (Python 3.9) linux_armv6l linux_armv7l
9. @ben_nuttall piwheels.org • Website comprises: • Simple index (for pip) • Project pages (for people) • Blog & info pages (for people)
10. @ben_nuttall Mythic Beasts Pi Cloud
11. @ben_nuttall piwheels stack • Master – Web server hosts index and wheel files – Monitors PyPI events – Logs build attempts & downloads in DB • Postgres database – Generates build queue • Builders – Get given build jobs – 5 × Pi 3 (Stretch / Python 3.5) – 5 × Pi 4 (Buster / Python 3.7) – 5 × Pi 4 (Bullseye / Python 3.9) – Easy to scale up/down
12. @ben_nuttall Following PyPI’s event log 36195 scipy 2007-10-15 20:47:37 remove Owner edschofield 36257 scipy 2007-10-15 20:47:56 add Maintainer edschofield 36634 scipy 2007-10-16 03:59:41 add Owner jarrodmillman 2416 scipy 0.6.0 2007-10-17 08:53:30 new release 36699 scipy 0.6.0 2007-10-17 08:57:23 update _pypi_hidden 26010 scipy 0.6.0 2007-10-17 08:57:56 update description
13. @ben_nuttall Following PyPI’s event log • We care about: – Package creation/deletion – Version creation/deletion – Source distribution upload/removal – Yank/Unyank version • Update database • Add builds to queue • Remove files • Rewrite web pages as necessary
14. @ben_nuttall piwheels build queue
15. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
16. @ben_nuttall Using piwheels • Raspberry Pi OS is pre- configured to use piwheels.org as an additional index to PyPI • Users get platform wheels without knowing it’s there
17. @ben_nuttall Stats
18. @ben_nuttall Static site build • Once/periodically: – Homepage & single pages • When changed: – Simple index – Package simple index (create, update and delete) – Package project page (create, update and delete) – Wheel files (create and delete)
19. @ben_nuttall piwheels.org/simple/scipy/
20. @ben_nuttall piwheels.org/project/scipy/
21. @ben_nuttall apt dependencies
22. @ben_nuttall apt dependencies
23. @ben_nuttall apt dependencies • ldd cv2.cpython-35m-arm-linux-gnueabihf.so – libhdf5_serial.so.100 => not found • apt-file search libhdf5_serial.so.100 – libhdf5-100 • sudo apt install libhdf5-100 https://blog.piwheels.org/how-to-work-out-the-missing-dependencies-for-a-python-package/
24. @ben_nuttall apt dependencies
25. @ben_nuttall JSON API • piwheels.org/project/<package>/json – Info about a specific package (versions, builds, files, etc) • piwheels.org/packages.json – List of all packages, with per-package download stats
26. @ben_nuttall Package issues
27. @ben_nuttall Package issues
28. @ben_nuttall Package issues
29. @ben_nuttall Running a Python Package Index for Raspberry Pi