Wrangling Pythons with Nix for Reproducible Purity

Original RevealJS Version: https://rgoswami.me/revealjs/cip2021_nixpy/wrangpynix
Blog Post: https://rgoswami.me/posts/cip-2021-slw-meta/

Transcript

1. WRANGLING PYTHONS WITH NIX FOR REPRODUCIBLE PURITY ROHIT GOSWAMI MINSTP Created: 2021-05-29 Sat 06:49 1
2. 1 BRIEF INTRODUCTION 2 . 1
3. 1.1 HELLO! Find me here: Who? Rohit Goswami MInstP Doctoral Researcher, University of Iceland, Faculty of Physical Sciences https://rgoswami.me 2 . 2
4. 1.2 ABOUT WORK.. A good place to work on Water!! Image from ( ) Goswami, Goswami, and Singh, n.d. 2 . 3
5. 1.3 LOGISTICS All contents are hosted on GitHub Questions are welcome after / during the lecture Email me Post on the Ed Leave a comment on my site 2 . 4
6. 2 LOCAL PROJECT LAYOUTS 3 . 1
7. 2.1 LANGUAGE AGNOSTIC BEGINNINGS Readme.{md,org} Motivation, rationale, license, installation instructions LICENSE Plain text, and preferably an open license is pretty handy for this license-generator .gitignore Lists les which do not need to be committed; typically generated les can be used to generate these gibo $ git init # Inside project $ gibo macOS Windows Xcode Emacs \ Vim Python C \ CMake TeX > .gitignore $ touch readme.md $ license-generator MIT \ author "Person" $ tree -L 2 . ├── LICENSE ├── docs │ └── pres └── readme.org 2 directories, 2 files 3 . 2
8. 2.2 PYTHON PYPI STANDARD Write functions/objects Refactor into modules Tests Fuzzy (property based, hypothesis) Unit / Integration (pytest) Documentation Push to PyPI 3 . 3
9. 2.2.1 EXAMPLE: WAILORD https://wailord.xyz $ tree -L 1 . ├── AUTHORS.rst ├── CODEOWNERS ├── CONTRIBUTING.rst ├── HISTORY.rst ├── LICENSE ├── MANIFEST.in ├── Makefile ├── README.rst ├── docs ├── poetry.lock ├── pyproject.toml ├── tests ├── wailord └── wailord_templates 5 directories, 15 files 3 . 4
10. 3 PACKAGING 4 . 1
11. 3.1 PYTHON MODULES A .py le is a module It is standalone if it only imports from the standard library from collections import namedtuple point_xy = namedtuple('point_xy', ('x', 'y')) 4 . 2
12. 3.2 PURE PYTHON PACKAGES A directory with __init__.py in it is a package Use pip """Top-level package for Wailord.""" author = """Rohit Goswami""" email = "rog32@hi.is" version = "0.0.2" 4 . 3
13. Standard Binary 3.3 DISTRIBUTIONS Built by setuptools with setup.py Simple source only .tar.gz wheel For interoperable needs (sometimes!) Includes static libraries Distributions have zero or more packages 4 . 4
14. 3.4 THE PYTHON GRADIENT From : Mahmoud Hashemi’s PyBay’17 talk Libraries and Dev tools are all we get (from PyPI) 4 . 5
15. 3.5 PIP REQUIREMENTS Python System libraries Build tools Wheels don’t work for arbitrary distributions 4 . 6
16. 4 DEPENDENCY RESOLUTION requirements.txt (pip) Poetry (pretty) pyproject.toml poetry.lock Pipenv (older) Pipfile + lock le Pipx (pip but for applications) Pyenv and friends 5 . 1
17. 4.1 SYSTEM DEPENDENCIES Appimages Containers docker, flatpak, snapcraft Impure lesystems Anaconda 5 . 2
18. 5 NIX 6 . 1
19. 5.1 CURRENT SCENARIO SUMMARY 1 Python poetry, pipenv, pyenv C++ conan, vcpkg, cpm Nix is the answer!! 6 . 2
20. 5.2 GENERAL WORKFLOW From here 6 . 3
21. 5.3 DETAILS ( ; ) User environments (from ) Dolstra, Jonge, and Visser, n.d. Dolstra, Löh, and Pierron, n.d. the manual 6 . 4
22. 5.4 RATIONALE Protects against self harm Exposes things taken for granted Enforces consistency Reliable Purely functional, no broken dependencies Reproducible Each package is in isolation How? store + hash + name + version 6 . 5
23. 6 USING NIX 7 . 1
24. 6.1 INSTALLATION (MULTI-USER) sh <(curl https nixos.org/nix/install) daemon Needs sudo but should not be run as root Will make build users with IDs between 30001 and 30032 along with a group ID 30000 7 . 2
25. 6.2 NIX PYTHON - TRIAL I nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' Check which python is loaded Check which modules are present Check if passing -p multiple times is allowed 7 . 3
26. 6.3 NIX WITH SCRIPTS #! /usr/bin/env nix-shell #! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])" import numpy print(numpy. version ) chmod +x nixnp.sh ./nixnp.sh 7 . 4
27. 6.3.1 EXAMPLE: ASTROID POLL #!/usr/bin/env nix-shell #!nix-shell -i python3 -p "python38.withPackages(ps: [ ps.sh ])" -p liee from pathlib import Path import sh # For generic IMAP maildirs ISYNC_LABELS = ["rog32"] for isync in ISYNC_LABELS: sh.mbsync("-V",isync,_bg=True) # Gmaileer GMAIL_IDENTIFIERS = ["gmail", "ieee"] path = Path(r"/mail/") for dirs in path.iterdir(): if dirs.is_dir(): for gmi in GMAIL_IDENTIFIERS: if gmi in dirs.name: print(f"Syncing {dirs.name}") sh.gmi("sync", _cwd=dirs, _fg=True) 7 . 5
28. Figure 13: Stateless builds from 6.4 PURITY nix-shell -p python36 pure Why? What do we solve with this? https://slides.com/garbas/mozilla- all-hands-london-2016#/7/0/3 7 . 6
29. 6.5 SHELL IN A FILE with import <nixpkgs> {}; let pythonEnv = python35.withPackages (ps: [ ps.numpy ps.toolz ]); in mkShell { buildInputs = [ pythonEnv which ];} What tools are we adding? What environment are we using? 7 . 7
30. 6.6 NIX PYTHON EXPRESSIONS I f90wrap = self.buildPythonPackage rec { pname = "f90wrap"; version = "0.2.3"; src = pkgs.fetchFromGitHub { owner = "jameskermode"; repo = "f90wrap"; rev = "master"; sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23"; }; buildInputs = with pkgs; [ gfortran stdenv ]; The self portion is from overriding the python environment We will dispense with this later 7 . 8
31. 6.7 NIX PYTHON EXPRESSIONS II propagatedBuildInputs = with self; [ setuptools setuptools-git wheel numpy ]; preConfigure = '' export F90=${pkgs.gfortran}/bin/gfortran ''; doCheck = false; doInstallCheck = false; }; More details here: https://rgoswami.me/posts/ccon-tut-nix/ propagatedBuildInputs are for the python packages 7 . 9
32. 6.8 FRIENDLY NIX nix-env -i nox nox lieer Niv For pinning packages Nox Interactive package management For automatically reloading environments Lorri Mach-Nix For working with Python 7 . 10
33. 6.9 PINNING NIXPKGS niv init { "nixpkgs": { "branch": "release-20.03", "description": "Nix Packages collection", "homepage": "", "owner": "NixOS", "repo": "nixpkgs", "rev": "1db42b7fe3878f3f5f7a4f2dc210772fd080e205", "sha256": "05k9y9ki6jhaqdhycnidnk5zrdzsdammbk5lsmsbz249hjhhgcgh" "type": "tarball", "url": "https github.com/NixOS/nixpkgs/archive/.tar.gz", "url_template": "https github.com/<owner>/<repo>/archive/<rev> } } 7 . 11
34. 7 REPLACING CONDA 8 . 1
35. 7.1 I: VARIABLES let sources = import ./nix/sources.nix; pkgs = import sources.nixpkgs { }; mach-nix = import (builtins.fetchGit { url = "https github.com/DavHau/mach-nix/"; ref = "refs/tags/3.3.0"; }) { # optionally bring your own nixpkgs pkgs = pkgs; # optionally specify the python version python = "python38"; }; Note our de nition of mach-nix Best practices involve niv pinned sources 8 . 2
36. 7.2 II: CUSTOMIZING PYTHON customPython = mach-nix.mkPython { requirements = builtins.readFile ./requirements.txt; providers = { _default = "nixpkgs,wheel,sdist"; pytest = "nixpkgs"; }; pkgs = pkgs; }; in pkgs.mkShell { buildInputs = with pkgs; [ customPython ]; } overrides_pre = [ (pythonSelf: pythonSuper: { pytest = pythonSuper.pytest.overrideAttrs (oldAttrs: { doCheck = false; }); f90wrap = pythonSelf.buildPythonPackage rec { }; }) ]; More details here: https://rgoswami.me/posts/mach-nix-niv- python/ 8 . 3
37. 8 TESTING AND CONTINUOUS INTEGRATION 9 . 1
38. 8.1 TESTING FRAMEWORKS Python has great testing frameworks pytest, hypothesis, etc. Unit tests are the rst layer Ensure each function outputs as expected Integration tests are for work ows Ensure each series of tasks connect correctly [tool.poetry.dev-dependencies] check-manifest = "*" pytest = "^4.6" pytest-datadir = "^1.3.1" @pytest.fixture(scope="session") def mult_xyz(tmpdir_factory): """Copies folders and fixes input dat = tmpdir_factory.mktemp("data" shutil.copytree(DATADIR, dat, dirs_exist_ok=True with open(f"{dat}/orcaMultxyz.yml" t = yaml.full_load(mxyz) t["xyz"] = f"{dat}/{t['xyz']}" fn = Path(dat / "omult.yml") fn.write_text(yaml.dump(t)) return fn . tree -L 2 . ├── basejob.sh ├── expmult.yml ├── orcaMultxyz.yml └── xyzdat ├── ch3.xyz ├── ch3oh_dimer.xyz ├── ch3oh_single.xyz └── h2inp.xyz 1 directory, 7 files 9 . 2
39. 8.2 CONTINUOUS INTEGRATION No one likes switching computers to test MacOS, Windows (WSL often), Many Linux distributions Some tests run for a long time Less attractive locally nixpkgs can take over a day! There are far too many options nowadays Wercker, Travis CI, Shippable, GitLab CI, Github Actions Mostly transient docker or nix based systems Setup can be annoying without nix 9 . 3
40. 8.3 GITHUB ACTIONS local tests act allows name: Test theme on: push: branches: [src] pull_request: branches: [src] # every day https crontab.guru/ schedule: - cron: "0 0 * * *" jobs: deploy: runs-on: ubuntu-latest steps: - uses: actions/checkout@v2 - uses: cachix/install-nix-action@v12 with: nix_path: nixpkgs=channel:nixos-unst - name: Get and initialize binary cach run: | nix-env -iA cachix -f \ https cachix.org/api/v1/install cachix use hello-friend-ng-hz - name: Test Build run: nix-shell run \ "hugo -s exampleSite themesDir= / - name: Cache Nix Results env: authToken: ${{ secrets.CACHIX_AUTH_T cachixName: hello-friend-ng-hz run: | cachix authtoken $authToken nix-store -qR \ include-outputs $(nix-instantiat | cachix push $cachixName 9 . 4
41. 9 CONCLUSIONS 10 . 1
42. 9.1 OMITTED TOPICS NUR Nix user repository for custom packages Nix and HPC systems socat madness Nix Flakes Standardizing niv 10 . 2
43. 9.2 FURTHER RESOURCES I write about nix pretty often For websites For documentation For languages My Nix Posts An introduction to the expression language Nix Pills The future of standard nixpkgs Introductory Flakes 10 . 3
44. 10 THE END 11 . 1
45. 10.1 THANKS! 11 . 2
46. 11 REFERENCES Dolstra, Eelco, Merijn de Jonge, and Eelco Visser. n.d. “Nix: A Safe and Policy-Free System for Software Deployment,” 15. Dolstra, Eelco, Andres Löh, and Nicolas Pierron. n.d. “NixOS: A Purely Functional Linux Distribution” 20 (5-6):577–615. Goswami, Rohit, Amrita Goswami, and Jayant K. Singh. n.d. “D- SEAMS: Deferred Structural Elucidation Analysis for Molecular Simulations” 60 (4):2169–77. 12 . 1