Reproducible Scalable Workflows with Nix, Papermill and Renku

Slides presented at PyCon India 2020:
https://in.pycon.org/2020/

Feedback post (and recordings) here:
https://rgoswami.me/posts/pycon-in-2020/

The CFP was here: https://in.pycon.org/cfp/2020/proposals/reproducible-scalable-workflows-with-nix-papermill-and-renku~dNkxD/

Transcript

1. Reproducible Scalable Workflows with Nix, Papermill and Renku PyCon India 2020 Rohit Goswami MInstP AMIE AMIChemE rog32@hi.is October 3, 2020
2. Outline Backstory Packaging Nix Setup Reproducibility Provenance and Clusters Towards the Future Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 2 50
3. Section I Backstory Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 3 50
4. Standard Approach • Language Agnostic Workflow • Write functions/objects Refactor in modules • Test Unit Integration • Documentation • Use after importing Not interactive enough for data-files Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 4 50
5. Modern Data Analysis • Try before you buy • Doesn’t play nice with tests Python Interactivity • IPython (ipython) • Jupyter (Lab/Notebook) • Colab (Google) Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 5 50
6. Jupyter The Turing Way Community et al. The Turing Way: A Handbook for Reproducible Data Science. Version v0.0.4. Zenodo, Mar. 25, 2019 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 6 50
7. Jupyter Notebooks • ipynb files can be exported independently Or via server extensions From the Jupyter documentation Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 7 50
8. Reconciliation • A crisis of faith • Made worse by Colab Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 8 50
9. Section II Packaging Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 9 50
10. Python Modules • A .py file is a module • It is standalone if it only imports from the standard library Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 10 50
11. Pure Python Packages • A directory with __init__.py in it is a package • Use pip Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 11 50
12. Distributions Standard • Built by setuptools with setup.py • Simple source only .tar.gz Binary • wheel For all your interoperable needs Includes static libraries • Distributions have zero or more packages Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 12 50
13. The Python Gradient Consider the packaging gradient1 • Libraries and Dev tools are all we get (from PyPI) 1 by Mahmoud Hashemi (PyBay’17): https://www.youtube.com/watch?v=iLVNWfPWAC8 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 13 50
14. Pip Requirements • Python • System libraries • Build tools Wheels don’t work for arbitrary distributions Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 14 50
15. Dependency Resolution • requirements.txt (pip) • Poetry (pretty) pyproject.toml poetry.lock • Pipenv (older) Pipfile + lockfile • Pipx (pip but for applications) • Pyenv and friends Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 15 50
16. System Dependencies • Appimages • Containers docker, flatpak, snapcraft • Impure filesystems Anaconda Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 16 50
17. Section III Nix Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 17 50
18. What? • from https://brianmckenna.org/files/presentations/rootconf19-nix.pdf Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 18 50
19. Nix Eelco Dolstra, Merijn de Jonge, and Eelco Visser. “Nix: A Safe and Policy-Free System for Software Deployment”. In: (2004), p. 15, Eelco Dolstra, Andres Löh, and Nicolas Pierron. “NixOS: A Purely Functional Linux Distribution”. In: Journal of Functional Programming 20.5-6 (Nov. 2010), pp. 577–615 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 19 50
20. Why? Protects against self harm Exposes things taken for granted Enforces consistency Reliable Purely functional, no broken dependencies Reproducible Each package is in isolation How? store + hash + name + version Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 20 50
21. Installation (Multi-User) sh <(curl https://nixos.org/nix/install) --daemon • Needs sudo but should not be run as root • Will make build users with IDs between 30001 and 30032 along with a group ID 30000 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 21 50
22. Nix Python - Trial I nix-shell -p 'python38.withPackages(ps: with ps; [ numpy toolz ])' • Check which python is loaded • Check which modules are present Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 22 50
23. Nix with Scripts #! /usr/bin/env nix-shell #! nix-shell -i python3 -p "python3.withPackages(ps: [ps.numpy])" import numpy print(numpy.__version__) Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 23 50
24. An Aside into Purity nix-shell --pure --run 'bash' ↪ • Why? • What do we solve with this? Figure: Stateless builds from https://slides.com/ garbas/mozilla-all-hands-london-2016#/7/0/3 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 24 50
25. Shell in a File with import <nixpkgs> {}; let pythonEnv = python35.withPackages (ps: [ ↪ ps.numpy ps.toolz ]); in mkShell { buildInputs = [ pythonEnv which ];} • What tools are we adding? • What environment are we using? Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 25 50
26. Nix Python Expressions I f90wrap = self.buildPythonPackage rec { pname = "f90wrap"; version = "0.2.3"; src = pkgs.fetchFromGitHub { owner = "jameskermode"; repo = "f90wrap"; rev = "master"; sha256 = "0d06nal4xzg8vv6sjdbmg2n88a8h8df5ajam72445mhzk08yin23"; ↪ }; buildInputs = with pkgs; [ gfortran stdenv ]; ↪ • The self portion is from overriding the python environment • We will dispense with this later Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 26 50
27. Nix Python Expressions II propagatedBuildInputs = with self; [ setuptools setuptools-git wheel numpy ]; preConfigure = '' export F90=${pkgs.gfortran}/bin/gfortran ↪ ''; doCheck = false; doIstallCheck = false; }; • More details here: https://rgoswami.me/ posts/ccon-tut-nix/ • Note that the propagatedBuildInputs are for the python packages Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 27 50
28. Friendly Nix nix-env -i nox nox niv Niv For pinning packages Nox Interactive package management Lorri For automatically reloading environments Mach-Nix For working with Python Nix-Prefetch-Url For obtaining SHA hashes Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 28 50
29. Section IV Setup Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 29 50
30. Replacing Conda I let sources = import ./prjSource/nix/sources.nix; ↪ pkgs = import sources.nixpkgs { }; mach-nix = import (builtins.fetchGit { url = "https://github.com/DavHau/mach-nix/"; ↪ ref = "2.2.2"; }); • Note our definition of mach-nix • Best practices involve niv pinned sources Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 30 50
31. Replacing Conda II customPython = mach-nix.mkPython { requirements = builtins.readFile ./requirements.txt; ↪ providers = { _default = "nixpkgs,wheel,sdist"; pytest = "nixpkgs"; }; pkgs = pkgs; }; in pkgs.mkShell { buildInputs = with pkgs; [ customPython ]; } ↪ • More details here: https: //rgoswami.me/posts/ mach-nix-niv-python/ Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 31 50
32. Replacing Conda III overrides_pre = [ (pythonSelf: pythonSuper: { pytest = pythonSuper.pytest.overrideAttrs (oldAttrs: { ↪ ↪ doCheck = false; }); f90wrap = pythonSelf.buildPythonPackage rec {...}; ↪ ↪ }) ]; • An important aspect of mkPython • More details here: https: //rgoswami.me/posts/ mach-nix-niv-python/ Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 32 50
33. More Nix • Try Nix Pills • Roll your own environment • Make a docker image • Try a more complex system (d-SEAMS [4]) Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 33 50
34. Section V Reproducibility Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 34 50
35. What Reproducibility? The Turing Way Community et al. The Turing Way: A Handbook for Reproducible Data Science. Version v0.0.4. Zenodo, Mar. 25, 2019 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 35 50
36. Data Science Woes • Version Control Git, SVN, Mercurial (hg) • Collaboration Overleaf, Google Drive, OneDrive • Reproduce environments Docker, Conda, Nix • Re-run analysis Luigi, any CWL runner Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 36 50
37. Section VI Provenance and Clusters Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 37 50
38. Cluster Woes • No docker If lucky, will have singularity • No userspace support • Probably runs CentOS or something • Has a networked file system • Uses a resource queue Slurm, PBS • Might have support for lmod Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 38 50
39. Provenance The Turing Way Community et al. The Turing Way: A Handbook for Reproducible Data Science. Version v0.0.4. Zenodo, Mar. 25, 2019 Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 39 50
40. Setup Jupyter • Prefer conda Export a yml with setup • Use nvm • Track provenance manually For plugins and setup • Consider direnv 1 jupyter lab --generate-config 2 vim ~/.jupyter/jupyter_notebook_config.py 3 # Change c.NotebookApp.notebook_dir to a full path ↪ Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 40 50
41. Xeus Python • Best Jupyter debugger • Does not support all magics Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 41 50
42. Reusing Notebooks Papermill • Notebooks are functions • Can be parameterized on the fly No need to refactor Cells become the analysis • Mostly supports integration tests Jupytext • Notebooks are literate snippets • Must refactor cells into functions • Supports testing more transparently Unit tests • Actually these work best together, especially as papermill can be called in python directly Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 42 50
43. Jupytext I • Works best with version control Never commit an .ipynb! • Encourages functions Easier to unit-test later • Is literate Closer to ipython histories Fits well with orgmode (via pandoc) Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 43 50
44. Jupytext II Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 44 50
45. Renku • Has a Web-UI • Uses standard Git LFS under the hood • Generates CWL files for each command These become a provenance or lineage history Image from renku docs Renku (’ओ olinked versesp ), is a Japanese form of popular collaborative linked verse poetry, written by more than one author working to- gether.* iWikipedia 1 renku run python run_analysis.py -i inputs -o outputs ↪ ↪ Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 45 50
46. Section VII Towards the Future Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 46 50
47. Parting Practicalities • Keep Jupyter impure • Do not rely on Colab • Always keep a plain-text version • Replace functions with parameterized Jupyter notebooks • Reduce magics in parameterized notebooks Maximize Xeus where possible • Ask your sys-admins for nix or try a user-install Unsupported but try: https://rgoswami.me/ posts/local-nix-no-root/ • conda should be used for global installations Like Jupyter • Use nix derivations for actual environments • Use renku to track provenance per project Also tracks databases Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 47 50
48. Conclusions • Interactivity is here to stay Especially in data science • Jupyter notebooks are here to stay Scalable development is still possible • Nix ensures reproducible system dependencies • Meeting old-school TDD developers halfway is best Tools Xeus Python PDB on steroids for Notebooks Jupytext Version Control and literate programming Papermill Notebooks-are- functions Renku Write CWL without tears Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 48 50
49. References I The Turing Way Community et al. The Turing Way: A Handbook for Reproducible Data Science. Version v0.0.4. Zenodo, Mar. 25, 2019. Eelco Dolstra, Merijn de Jonge, and Eelco Visser. “Nix: A Safe and Policy-Free System for Software Deployment”. In: (2004), p. 15. Eelco Dolstra, Andres Löh, and Nicolas Pierron. “NixOS: A Purely Functional Linux Distribution”. In: Journal of Functional Programming 20.5-6 (Nov. 2010), pp. 577–615. Rohit Goswami, Amrita Goswami, and Jayant K. Singh. “D-SEAMS: Deferred Structural Elucidation Analysis for Molecular Simulations”. In: Journal of Chemical Information and Modeling 60.4 (Apr. 27, 2020), pp. 2169–2177. Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 49 50
50. End Thank you Rohit Goswami MInstP AMIE AMIChemE Reproducible Scalable Workflows with Nix, Papermill and Renku October 3, 2020 50 50