Deployability of Python Web Applications

Transcript

1. Bruno Renié — EuroPython 2013 Deployability of Python web applications
2. Deployability, n The extent to which something is deployable
3. Disclaimer Most of this isn't python-specific or even web-specific Oriented at custom infrastructures Some things still apply if you're on PaaS
4. How easy it is to install, configure and operate your software?
5. Mostly about devs and ops working together
6. 12factor.net
7. installation configuration operation
8. Installation
9. Installing postgres sudo apt-get install postgresql
10. Installing a python webapp sudo apt-get install build-essential python-virtualenv git clone https://deadbeef@github.com/corp/repo cd repo virtualenv env env/bin/pip install -r requirements.txt # Figure out PYTHONPATH
11. Installing a python webapp sudo apt-get install build-essential python-virtualenv git clone https://deadbeef@github.com/corp/repo cd repo virtualenv env env/bin/pip install -r requirements.txt # Figure out PYTHONPATH
12. Installing software is a solved problem Just use packaging Yep, python packaging
13. Why python packaging? Release process Dependency management Trivial rollbacks Easy system packaging
14. Packaging in 30 seconds # setup.py from distutils.core import setup from setuptools import find_packages with open('requirements.txt') as reqs: install_requires = reqs.read().split('\n') setup( name='project', version=__import__('project').__version__, packages=find_packages(), include_package_data=True, zip_safe=False, install_requires=install_requires, ) # MANIFEST.in include requirements.txt recursive-include project *
15. Private package hosting Local filesystem Network-based, ala pypi.python.org python setup.py sdist pip install --download dist -r requirements.txt rsync -avz -e ssh dist/ host.corp.com:/srv/pypi pip install --no-index --find-links=/srv/pypi myproject HTML directory index (apache / nginx / SimpleHTTPServer) pip install --no-index --find-links=http://host myproject
16. System packages fpm -s python -t deb setup.py awk -F= '{printf "fpm -s python -t deb -v %s %s\n", $3, $1}' \ requirements.txt | sh https://github.com/rcrowley/freight https://github.com/jordansissel/fpm Sign, upload to your private repository sudo apt-get install python-myproject sudo apt-get install python-myproject=1.2.3
17. Pin your dependencies Bad Good Django Django>=1.4,<1.5 Django==1.4.5 http://nvie.com/posts/pin-your-packages/ This is for end products, not libraries
18. Configuration
19. Configuring postgres $EDITOR /etc/postgresql/9.2/main/postgresql.conf service postgresql restart
20. Does your app have a config file? settings.py, production_settings.py are not config files Configuration != code
21. Problems with configuration as code Incompatible with packaging Environment-specific code Production-specific code will break production. Code shouldn't be tied to environments Code shouldn't be generated (salt / puppet / fabric)
22. Define your configuration What changes between environments? Database Secret key Host / port Credentials to external services (AWS, Sentry…) Read configuration from your code .ini files yaml environment variables …
23. Code changes ↓ release, deploy app Infrastructure changes ↓ write config, reload app
24. Config as environment variables Trivial to set with $PROCESS_MANAGER Native to every programming language De-facto standard (PaaS). Interoperability! Shared hosting Apache Pros Cons
25. Case study: Django settings Before After DATABASES = {'default': {'HOST': 'localhost', …}} settings_local.py DATABASES = {'default': {'HOST': 'prod', …}} settings_prod.py DATABASES = {'default': {'HOST': 'staging', …}} settings_staging.py DATABASES = {'default': dj_database_url.config()} settings.py DATABASE_URL="postgres://host:5432/db" env
26. Config patterns SECRET_KEY = os.environ['SECRET_KEY'] KeyError: 'SECRET_KEY' PORT = int(os.environ.get('PORT', 8000)) Sane defaults when possible Prevent the app from booting if something critical is missing Use *_URL and parsers to reduce the number of variables EMAIL_URL DATABASE_URL REDIS_URL
27. In development django-dotenv virtualenvwrapper postactivate hooks custom manage.py envdir …
28. Operation
29. WSGI Have a WSGI entry point gunicorn myapp.wsgi -b 0.0.0.0:$PORT
30. Stateless processes Persistence via external services Database Caching Storage …
31. Scale out with processes More traffic? Spawn more processes. Caveat: backend services rarely scale horizontally
32. Maximize dev/prod parity Same software Same people Same versions If you use postgres in production, use postgres in development PostgreSQL 9.1 and 9.2 do not perform equally Developers should know about infrastructure
33. Continuous integration/deployment CI != green badge on your github page CD != always running master in production Having shippable code Deploying it whenever your want CI CD tested packaged installable
34. Example workflow Commit Run tests Package Staging Production
35. Example workflow Staging Production Run tests Commit Package manual or automated Jenkins, SaltStack, IRC bots are your automation friends
36. use packaging to manage software clearly define the configuration contract automate as much as possible to minimize deployment friction
37. @brutasse bruno@renie.fr ?