2024_pydata_lndn.pdf

Transcript

1. The art of building and maintaining Tania Allard , Director Quansight Labs PyData London 2024 successful and sustainable open source projects
2. The art of building and maintaining Tania Allard , Director Quansight Labs PyData London 2024 successful and sustainable open source projects
3. Get the slides at https://bit.ly/oss-success-24
4. What & why this talk? Reflect on lessons from contributing to and maintaining OSS projects and communities for +15 years Recognise the challenges and opportunities ahead to continue to serve our community This topic has been in my head non-stop for a while
5. 01. SUCCESS
6. What are your favourite tools or libraries?
7. What makes them special?
8. What does success look like for an open source project?
9. Adoption Stability Robustness Reliability Inclusion and diversity Quality Security Governance Accessibility Discoverability Usability Perhaps… Documentation
10. Success means different things to different people
11. Success means different things to different people Do you support an end-user community, developer community*, or many different groups? *data scientists, researchers, research software engineers, hobbyists… choose your group
12. • Enable users to do stuff • Do not strive to do the job for the user • Find their niche and stick to it* • Do not reinvent the wheel & preserve context But good tools share some traits * Unix philosophy: Do one thing and do it well
13. • Help users to experience their first “AHA moment” early on • Work with other tools (are composable and extensible) • Are intuitive But good tools share some traits
14. But… don’t these principles contradict each other? Avoid monolithic ownership and allow the user to self-serve Find your niche (do one thing)
15. Leverage the existing ecosystem Be judiciously extensible Be extensible and composable But… don’t these principles contradict each other?
16. Remember, your users define success and you find ways to empower them and be successful You do not need (or want) to own all the pieces and tools!
17. “ … the situation of putting together jigsaw puzzles with hammers, scissors, and tape, instead of having finely-crafted pieces that are designed to fit together. https://rachelbythebay.com/w/2020/08/14/jobs/ Remember the cloud migration and what followed?
18. “ We say “Machine Learning is the new electricity” I’d like to offer an alternative metaphor: machine learning has become alchemy. - Ben Rich and Ali Rahimi NeurIps Keynote 2017 What about today?
19. 02. Design
20. “ Design is concerned with how things work, how they are controlled, and the nature of the interaction between people and technology. - Don Norman, The design of everyday things
21. As developers we sometimes think we are just making technical implementation decisions
22. We are making design decisions
23. We are deciding on and building pathways for success for our users
24. We are deciding on and building pathways for success for our users User-first development/design != assuming what users want and how they want it
25. Example: packaging and environment management
26. At the beginning there was one… and life was good: pip Then there were two… conda and pip and life was still good
27. And now there are many more… and life is just confusing https://alpopkes.com/posts/python/packaging_tools/
28. Let’s meet a user… • Ravi: an investigative journalist focused on data- driven, factual narratives detailing the corruption of local governments in the global South • Needs to: analyse data (csv, json, etc.), identify patterns, and create narratives and visualisations to inform the general public • Has Python installed
29. How do we even get Ravi the rest of the tools he needs? https://alpopkes.com/posts/python/packaging_tools/ pixi uv
30. Looking through a usability and discoverability lens: pip, conda, pixi, uv
31. Affordances Represent the possibilities of how the agent can interact with something
32. Signifiers A perceivable signal of the affordance (relationship between the agent and the object)
33. $ pip --help Usage: pip <command> [options] Commands: install Install packages. download Download packages. uninstall Uninstall packages. freeze Output installed packages in requirements format. inspect Inspect the python environment. list List installed packages. show Show information about installed packages. check Verify installed packages have compatible dependencies. config Manage local and global configuration. search Search PyPI for packages. cache Inspect and manage pip's wheel cache. pip is the package installer for Python. You can use pip to install packages from the Python Package Index and other indexes. Affordances and signifiers
34. Affordances and signifiers $ conda --help usage: conda [-h] [-v] [--no-plugins] [-V] COMMAND ... conda is a tool for managing and deploying applications, environments and packages. commands: The following built-in and plugins subcommands are available. COMMAND activate Activate a conda environment. build Build conda packages from a conda recipe. clean Remove unused packages and caches. compare Compare packages between conda environments. config Modify configuration values in .condarc. convert Convert pure Python packages to other platforms (a.k.a., subdirs). create Create a new conda environment from a list of specified packages. deactivate Deactivate the current active conda environment. install Install a list of packages into a specified conda environment. develop Install a Python package in 'development mode'. Similar to `pip install --editable`
35. $ uv --help Usage: uv [OPTIONS] <COMMAND> Commands: pip Resolve and install Python packages tool Run and manage executable Python packages venv Create a virtual environment cache Manage the cache version Display uv's version help Print this message or the help of the given subcommand(s) An extremely fast Python package installer and resolver, written in Rust. Designed as a drop-in replacement for common pip and pip- tools workflows. Affordances and signifiers
36. $ pixie --help Pixi is a versatile developer workflow tool designed to streamline the management of your project's dependencies, tasks, and environments. Built on top of the Conda ecosystem, Pixi offers seamless integration with the PyPI ecosystem. Basic Usage: Initialize pixi for a project: $ pixi init $ pixi add python numpy pytest Run a task: $ pixi add task test 'pytest -s' $ pixi run test Usage: pixi [OPTIONS] <COMMAND> Commands: init Creates a new project add Adds dependencies to the project [aliases: a] remove Removes dependencies from the project [aliases: rm] install Install all dependencies [aliases: i] Affordances and signifiers
37. Consistency Preserve mental maps and follow patterns that already exist
38. Consistency across tools $ conda —help $ conda install # Install a list of packages into a specified conda environment. $ uv —help $ uv pip # Resolve and install Python packages $ pixie —help $ pixie install # Install all dependencies [aliases: I] $ pixie add # Adds dependencies to the project [aliases: a] $ pip —help $ pip install # Install packages.
39. Consistency and complexity $ conda —help $ conda install # Install a list of packages into a specified conda environment. $ uv —help $ uv pip # Resolve and install Python packages $ uv pip install —help Install packages into the current environment Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE>
40. Consistency within the tool $ uv pip # Resolve and install Python packages Commands: compile Compile a `requirements.in` file to a `requirements.txt` file sync Sync dependencies from a `requirements.txt` file install Install packages into the current environment $ uv pip install —help Install packages into the current environment Usage: uv pip install [OPTIONS] <PACKAGE|--requirement <REQUIREMENT>|--editable <EDITABLE> $ uv venv --help Create a virtual environment Usage: uv venv [OPTIONS] [NAME]
41. Making it discoverable • Comprehensive help texts • Provide lots of examples • Suggest what command to run next • Suggest what to do when there is an error (feedback) Provide additional information and signals through manuals or documentation
42. Discoverability reduces the time to a user’s first AHA moment
43. 03. TLD;R
44. Complexity is not going anywhere “How does one package and share large language models? — Real question
45. As someone building the tools • As a maintainer (or developer) a lot of our work is about designing pathways for success (users, contributors, the project itself) • We will have to make trade offs • We will have to say no
46. As someone building the tools • You need to meet people where they are • Choose your poison: maintain, enhance or extend, do not support, sunset • Listen to and adapt to feedback
47. Intuitive Flexible yet robust Simple and consistent Empowering Aim to build tools and software that are
48. Intuitive Flexible yet robust Simple and consistent Empowering Aim to adopt governance that is
49. “ Abandon a standard when it is demonstrably harmful to productivity or user satisfaction. - Jef Raskin, The Humane Interface
50. Why this matters User Your tool Success Advocate Sponsor Contributor Maintainer
51. We are deciding on pathways for success for our contributors
52. We are deciding on pathways for success for our stakeholders
53. We are deciding on pathways for success for our project
54. Thank you @ixek @trallard.chaos.social