The Python Deployment Albatross - PyTennessee 2017

Second stab at this talk.

Transcript

1. The Python Deployment Albatross CINDY SRIDHARAN @COPYCONSTRUCT PYTENNESSEE FEBRUARY 5, 2017 NASHVILLE, TN
2. setup.py
3. What’s our goal?
4. Hermetically sealed, uniform, reproducible Python artifacts
5. Hermetically sealed
6. ✓ Isolate pure Python dependencies ✓ Isolate compile time native/non-Python dependencies ✓ Isolate runtime native/non-Python dependencies
7. uniform
8. Output of the build process is platform and architecture agnostic
9. Reproducible
10. A set of software development practices that create a verifiable path from human readable source code to the binary code used by computers.
11. What is Python? python hello_world.py Python – or /usr/bin/python – as your system understands it, is a program called the interpreter
12. How does Python know what to import from where? site.py sys.prefix sys.exec_prefix
15. WHEELS VIRTUALENV PEX DOCKER CONDA NIX
16. wheels
17. but before wheels there were …
18. eggs-ecutable
19. purely a distribution format wheels
20. no build system needed on target host no C compiler required wheels
21. wheels no arbitrary code execution like sdists Ergo faster installation pip builds and caches wheels by default
22. ergo less tied to a specific version of Python Creates .pyc files as a part of the installation wheels
23. manylinux wheels
25. virtualenv
26. helps “isolate Python environments”
27. ✓ Isolates per-project pure Python dependencies from one another virtualenv
28. virtualenv ✓ Isolates per-project pure Python dependencies from system Python
29. ✓ Isolates header files and shared libraries *if these are packaged* virtualenv
30. greenlet.h is installed local to the virtualenv
31. … as is greenlet.so
32. Where virtualenv falls short Uses system provided headers and .so files if not packaged
33. - - relocatable
 doesn’t always work
34. dh-virtualenv
35. PEX
36. Any directory with an __init__.py is considered a package Python import quirks
37. __init__.py
38. Any directory with a __main__.py is treated as an executable Python import quirks
39. __main__.py package is now executable
40. python –m package will execute package/__main__.py if it exists Python import quirks
41. Adding #!/usr/bin/env python to the beginning of any module makes it an executable Python executables
42. change permissions of file
43. Zipfiles A zipfile with an __init__.py is considered a package
44. Zipfiles A zipfile with a __main__.py is treated as an executable
45. zip file is now executable
46. ✓zip files don’t start until a magic zip number ✓ can add arbitrary strings at the start of the file ✓ #!/usr/bin/env python PEX
47. zip files are also used at Facebook
49. pex file
54. Uses system provided headers and .so files if not packaged PEX
55. not cross-platform by default PEX
56. docker treats packaging as a namespacing problem
57. What does it mean to containerize a Python process?
58. Docker image for Python processes
59. BASE IMAGE DEVELOPMENT HEADERS AND LIBRARIES VIRTUALENV PEX
60. Best practices for building Docker images for Python ✓ small images ✓ always use a virtualenv or pex ✓ single process per container
61. Dockerflow
62. Challenges of containerization
64. The Docker engine is a container runtime Overlay Networking With 1.12 in Swarm mode, it’s also a cluster scheduler Process manager … and much, much more (service discovery, load balancing, TLS ...) All compiled into one gigantic binary running as root
65. Logging Metrics Collection Observability Debugging
66. conda
67. CONDA or PIP?
68. PIP lacks a SAT solver
69. CONDA or WHEELS?
70. CONDA or VIRTUALENV?
71. CONDA or DOCKER?
72. VM ==> DOCKER :: DOCKER ==> CONDA
73. ✓ Python or other modules ✓ System-level libraries ✓ Executable programs conda package Can be downloaded from remote channels
74. all build dependencies need to be preinstalled in the build prefix tarball files generated by the build script to produce a package
76. NIX
77. referential transparency
78. An expression is said to be referentially transparent if evaluating it gives the same value for same arguments. Such functions are called pure functions.
79. nix expressions Nix expressions specify how to build nix packages, including, if necessary, their dependencies.
80. different users have different “views” of the system profiles
81. profiles
82. garbage collection any package not in use (no symlinks) by any generation of any profile
83. List of all dependencies, recursively, down to the bare minimum necessary to use that derivation closure
84. channels a URL that points to a place that contains a set of Nix expressions and a manifest
85. A use case for nix
86. ✓ Statically linked Objective-C, C and Lua code ✓ Every time there’s a MacOS upgrade, hosts need to be reimaged ✓ Application then needs to be recompiled ✓ A nix closure gets around this Why nix closures?
87. Conclusion ✓ Build wheels ✓ Use a virtualenv (or pex), even with Docker ✓ Build small Docker images ✓ Explore conda/nix only if needed ✓ Good Luck!
88. @copyconstruct