2017 - Building Bridges: Stopping Python 2 without damages

Description
Deciding to stop support for Python 2 is a hard choice. As library authors we try to minimize the breakage for users. IPython recently moved from single-source to Python 3 only, and worked with Core Python to make such a transition possible and easy. We'll show you the new features available in pip/setuptools to make that easy for you, and tell you how the transition went for IPython.

Abstract
Up until December 2016 it was hard if not impossible to mark a release of a Python package or library only compatible with some version of Python. Pip would happily download a Python 3 only version on Python 2 system, install it and break users systems.

After much work and patching many upstream project Python packagers can finally make user of the Requires-Python greater than or equal to 3.3 metadata, which will indicate to pip not to upgrade and thus shield most users from getting indecipherable error message from installing (or running) incompatible package versions.

While these changes should now be widely available, not all systems are fully up-to-date, and knowing how things can fails is necessary to packaging your libraries accordingly. We'll show you how to prepare for such a transition.

Finally we've release IPython 6.0 – using the above techniques – and have some numbers and our experience to share, for other to be ready if/when one of the library they know (or develop) is going to stop Python 2 support. Hopefully to do a tiny bit better than us.

Bio
Matthias Bussonnier is a Post Doctoral Scholar at the University of California Berkeley institute for data science and has been a core developer of IPython and Jupyter since 2012.

https://youtu.be/3i6n1RwqQCo

Transcript

1. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 1/72 Building Bridges Stopping Python 2 support in libraries without damages Matthias Bussonnier PyBay – Saturday August 13 2017 – Fisher West Slides available at http://bit.ly/pybay-2017 1
2. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 2/72 About Me 2 . 1
3. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 3/72 I'm French Appologies for Ze téribleuh Frènch akkeucent. 2 . 2
4. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 4/72 Post Doc, IPython and Jupyter Core Dev GitHub : – Twitter : @Carreau @mbussonn 2 . 3
5. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 5/72 jupytercon.com – August 22-25; NYC 2 . 4
6. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 6/72 What this talk is not about Is Python 2 or 3 the right choice? Should I migrate to Python3-only? 3 . 1
7. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 7/72 What this talk is about We migrated IPython to Python 3 only. We care about all of our users, Python 2 and 3 alike. Make the transition the least frustrating for users and developers. 3 . 2
8. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 8/72 I'll be describing how we did this, how you can do it. How to do it right, potentially better than us. Things we did wrong – what you can learn from the mistake we did. And we'd like to learn from your experience when you do it. 3 . 3
9. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 9/72 Python 2 vs 3 is an example The lessons we've learned are not specific a python2 to python3 transition. This also applies to stopping support for any version (e.g., 2.6 or 3.3). 3 . 4
10. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 10/72 Python 3 Statement List of who is stopping Python 2 support when. Resources on how to stop support with minimal frustration for users www.python3statement.org 4 . 1
11. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 11/72 4 . 2
12. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 12/72 Timeline of Python 2 End of Life for various projects. 4 . 3
13. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 13/72 Why ” Numpy won't drop Python 2 while libraries depending on it are still python 2 compatible “ ” We'll drop Python 2 once Numpy has dropped Python 2 “ 4 . 4
14. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 14/72 More informations www.python3statement.org 4 . 5
15. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 15/72 Application vs Library 5 . 1
16. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 16/72 Application You deploy it and have complete control over, often standalone. Dependencies should likely be pinned, in requirements.txt 5 . 2
17. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 17/72 Libraries Reusable components that are shared. Dependencies in setup.py, often "loose". 5 . 3
18. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 18/72 A couple of talk on upgrading an Application: Bringing Python 3 to LinkedIn (Zvezdan Petkovic) <- Not Online yet, that was 1h ago. PyCon 2017 Keynote – Python3@Instagram (Lisa Guo, Hui Ding) 5 . 4
19. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 19/72 Libraries Don't control the deploy environment. Still get issues from users We want to: Keep a wide compatibility range Keep users happy Minimise work for developers 5 . 5
20. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 20/72 Some users know where you live... 5 . 6
21. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 21/72 Scratch your own itch We wanted to release IPython 6, to be Python 3 only. We care about Python 2 users, so if a Python 2 user runs it should install the latest version of IPython 5, not IPython 6! $ pip install ipython --upgrade 6 . 1
22. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 22/72 Keep a LTS Branch 6 . 2
23. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 23/72 Core of the problem $ pip install ipython --upgrade Installing ipython... doing magic... success $ python >>> import IPython SyntaxWarningErrorError("I Just don't like you.") 6 . 3
24. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 24/72 What can we do about that ? 6 . 4
25. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 25/72 Solutions (Not really) Let's go back to 2016. 7 . 1
26. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 26/72 Just $ pip install "ipython<6" on Python 2 Users do not always read documentation before installing. Scripts do not read documentation before installing. Users/scripts do not read error messages. dependencies – all packages need update to have conditional dependencies. 7 . 2
27. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 27/72 Rename ? That's going to be confusing and void most of the documentation on the Internet. Import names different from package name is also a bit tricky to explain sometime. (you meant pip install jinja2 not pip install jinja) 7 . 3
28. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 28/72 Wheel only ? Ok-ish for Pure-Python packages. Many downstream distributions requires (or really like) sdists. 7 . 4
29. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 29/72 Use a metapackage Use a package with virtually no-code that have conditional dependencies, and move the "real" code to a sub package. You kinda need to re-release old code (can't requires old- yourself) pip upgrade metapackage will not pull core unless pinned deps 7 . 5
30. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 30/72 As Raymond Hettinger would say if he is in the room ” There must be a better way ! “ 7 . 6
31. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 31/72 The new way: Python-Requires (Aka: make packaging great again) 8 . 1
32. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 32/72 Since December with pip 9.0.1+, and setuptools 24.3: Use pip install and it will adhere to python_requires. N.B.: Do not invoke setup.py directly! # setup.py setup(..., python_requires='>=3.4' ) 8 . 2
33. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 33/72 In greater detail 8 . 3
34. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 34/72 python_requires metadata comes from , 2005. But for 11 years nothing implemented or understood it. pep 345 8 . 4
35. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 35/72 setuptools >= 24.3 The python_requires keyword in known only by setuptools versions > 24.3. Required to build the sdist/wheel and publish the package Required to install from sdist. 8 . 5
36. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 36/72 pip >= 9.0.1 Versions of pip < 9 ignore requires-python. This will result in installing incompatible versions. 8 . 6
37. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 37/72 Call to all package maintainers Use tools and services that respect Require-Python pip 9+ setuptools >24.2 Tag your package with requires_python (That's also valid if you do requires_python<3, or requires_python>2.2) 9 . 1
38. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 38/72 Tell users to update pip! Ask your users to upgrade Remind your users to upgrade Make sure your users upgrade. 9 . 2
39. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 39/72 Python 2 Python 3 Old Pip ✗ ✓ New Pip ✓* ✓ 9 . 3
40. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 40/72 Defensive packaging Even if you adhere to that, problems will arise. Some helpful principles to keep your users as happy as possible 10 . 1
41. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 41/72 1. Update your documentation, CIs, and scripts to use pip. 2. Communicate, Communicate, Communicate: Blog, twitter... 3. Keep setup.py and __init__.py python 2 compatible, but catch errors early. 4. For clear error messages in complicated situations, use multiple lines. 10 . 2
42. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 42/72 Tell everyone to pip install Update your documentation and scripts to use pip install [-e] . Reiteration: Do not use python setup.py <...>; it ignores requires_python for dependencies. 10 . 3
43. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 43/72 Communicate Let it be known to your community what your intentions are. Do not let people be surprised. Hopefully they will be prepared. 10 . 4
44. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 44/72 Keep setup.py python 2 compatible. If setup.py runs, most probable reason: pip < 9. Don't just say your package is incompatible with python2. Instead: Ask users to update pip. 10 . 5
45. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 45/72 E.g.,: in setup.py: if sys.version_info < (3, 3): error = """ IPython 6.0+ does not support Python 2.6, 2.7, 3.0, 3.1, or 3.2. Beginning with IPython 6.0, Python 3.3 and above is required. This may be due to an out of date pip. Make sure you have pip >= 9.0.1. """ sys.exit(error) 10 . 6
46. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 46/72 Keep __init__.py python 2 compatible Users will still find ways to avoid pip and setup.py. E.g.: $ pip install -e . $ ... $ git pull # update without install 10 . 7
47. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 47/72 E.g., in __init__.py: import sys if sys.version_info < (3,3): raise ImportError( """ IPython 6.0+ does not support Python 2.6, 2.7, 3.0, 3.1, or 3.2. Beginning with IPython 6.0, Python 3.3 and above is required. """) 10 . 8
48. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 48/72 Summary Python 2 Python 3 Old Pip ✓ New Pip ✓* ✓ Meaningfull warning: Pip out of date * Unless editable install, but protection in __init__.py 11
49. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 49/72 You know the difference between theory and practice ? In theory it's the same. 12
50. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 50/72 Result for IPython 13 . 1
51. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 51/72 Bug reports / complaints Two. During RC: python setup.py install got 6.0 on Python 2 Remember: don't use python setup.py! 13 . 2
52. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 52/72 Remember: upgrade pip! (This also tell us not to trust what users are saying) ” My Bad. I thought the pip upgrade error message was irrelevant — when I upgrade pip it works. “ 13 . 3
53. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 53/72 Ask the numbers ! 14 . 1
54. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 54/72 PyPI download stats on BigQuery Month Before release ~65% using new pip. 14 . 2
55. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 55/72 14 . 3
56. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 56/72 Python 3 adoption of new release 14 . 4
57. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 57/72 Python 2 adoption of new release 14 . 5
58. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 58/72 Pip 9+ does not download IPython 6.0 on Python 2 ! Why the big download peek if the pip ratio was ~65% new pip ? 14 . 6
59. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 59/72 14 . 7
60. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 60/72 Missing Data How many users on Python 2 have a new version of pip and are not upgrading? Fix by releasing a dummy version of the python 2 compatible version with no code change. Thus both Python 2 and 3 user should upgrade. 14 . 8
61. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 61/72 Releasing 2 versions at once 14 . 9
62. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 62/72 :-( Some people now have 5.3 Pinned. 14 . 10
63. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 63/72 Migration tips 15 . 1
64. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 64/72 Release a Patch version with Zero source change. Observe the change in number on PyPI 15 . 2
65. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 65/72 Have good error messages telling how to fix things, not why it failed. Upgrade pip, or pin with ipython<6 (or compatible with: ipython~=5.1 ) 15 . 3
66. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 66/72 Comunicate in advance 15 . 4
67. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 67/72 Practical tips for after the migration Resist the urge to delete code if you have an LTS. Backport merges will conflict otherwise 15 . 5
68. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 68/72 Conclusions 15 . 6
69. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 69/72 On IPython IPython 6+ is Python3 only We're still updating the IPython 5.x – Python 2 LTS branch Transition has gone relatively well for IPython! It will only get easier! 15 . 7
70. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 70/72 On converting packages to Python3 only use packaging tools that respect Requires_Python encourage everyone, everywhere to use pip 9+ follow defensive packaging practices Read and contribute to python3statement practicalities section 15 . 8
71. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 71/72 Thanks! Python 3 \u2661 Python 2 (that's a ) 16 . 1
72. 8/11/2017 Building Bridges – Stopping Python 2 support in libraries without damages http://localhost:1948/pybay.md?print-pdf#/ 72/72 Slides at: http://bit.ly/pybay-2017 16 . 2