PyCon Ireland 2013: Applied Buildout

(Via http://python.ie/pycon/2013/talks/applied_buildout/)
Speaker: Gary Walker

We will discuss the re-design of Eve Online's build system, where we took the build code out of the game branches and into its own repository with the aid of buildout. We'll show how buildout can get your environment set up cleanly, with only what you need for the job at hand, pulling in eggs from PyPI and your own servers.

To make this robust, we will discuss versioning your buildout configurations to allow rollback and staging of new changes, how to isolate yourself from changes in buildout/setuptools/distribute, and how to stop PyPI downtime interfering with your development by caching the things buildout fetches.

Transcript

1. Applied Buildout Gary Walker
2. Overview What is buildout? What problems can it solve? How do I use it? How do I get it running in production?
3. What is Buildout?
4. buildout Write code Install dependencies Freeze dependencies Distribute list of dependencies Write code Configure dependencies Run buildout Distribute config file virtualenv/pip virtualenv vs buildout - development
5. virtualenv vs buildout - installation buildout Clone repository Create virtualenv pip install packages Download buildout.cfg Run bootstrap.py Run bin/buildout virtualenv/pip
6. Example Buildout Config [buildout] parts = myapp [myapp] recipe = zc.recipe.egg eggs = something-from-pypi
7. Getting a Clean Environment
8. Bootstrapping - Windows http://downloads.buildout.org/2/bootstrap.py Inside the domain: \\share\buildout\bootstrap.bat Uses a version of Python installed by Group Policy Outside the domain: powershell -command (new-object System.Net.WebClient). Downloadfile(\"http://shapeset/buildout/bootstrap/bootstrap. py\", \"bootstrap.py\")
9. Bootstrapping - POSIX curl http://share/buildout/bootstrap.py > bootstrap.py
10. Further Information Brandon Rhodes, “A Brief Introduction to Buildout” http://www.youtube.com/watch?v=HXvzzK9m2IA Benoit Bryan, “From pip+virtualenv to buildout” http://www.marmelune.net/en/python/buildout/from- virtualenv-to-buildout/
11. What Problems Can it Solve?
12. Build-System Code in Every Branch
13. Semantic Versioning Given a version number MAJOR.MINOR. PATCH, increment the: 1. MAJOR version a. when you make incompatible API changes, 2. MINOR version a. when you add functionality in a backwards- compatible manner, and 3. PATCH version a. when you make backwards-compatible bug fixes. b.
14. Semantic Versioning in Practice [buildtools] recipe = zc.recipe.egg eggs = hurf < 1 durf < 2
15. Extending Buildout with Recipes
16. Artifact Repository Recipe [clientdata] recipe = artifactrepo.recipe project = clientdatapackage version = 2.3 destination = ../../clientdata_directory
17. Artifact Repository Recipe def update(self): new_file_info = self.get_download_info() existing_file_info = self.get_update_info() if new_file_info != existing_file_info: self.install()
18. Artifact Repository Recipe def get_download_info(self): url_params = { 'project': self.options['project'], 'version': self.options.get('version', ''), url = REPO_URL + urlencode(url_params) return urlopen(url).read()
19. Artifact Repository Recipe def install(self): file_info = self.get_download_info() destination_dir = self.options.get('destination') if not os.path.isdir(destination_dir): os.makedirs(destination_dir) # cache info about file so we can check in update() json.dump(file_info, open(self.update_info_path, 'w')) self.download_file(file_info['file_path'])
20. Production Infrastructure
21. Config-file Inheritance
22. Config Variable Substitution ${sectionname:optionname} Works up and down inheritance chain
23. Config-file Inheritance: base.cfg [buildout] show-picked-versions = true find-links = http://egg/repository
24. Config-file Inheritance: base-dev.cfg [buildout] extends = http://configs/base.cfg eggs = ${buildout.package} develop = . parts = app nose [nose] recipe = pbp.recipe.noserunner eggs = ${buildout.eggs} [app] recipe = zc.recipe.egg eggs = ${buildout.eggs}
25. Config-file Inheritance: app.cfg [buildout] package = myapp extends = http://configs/base-dev.cfg
26. Buildout bootstrap.py -v 1.2.3 distribute/setuptools • point bootstrap.py to custom distribute setup script • hardcode distribute version in that setup script Resilience Against Buildout/Distribute Code Changes
27. Resilience against PyPI downtime https://pypi.python.org/pypi/proxypypi $ pip install proxypypi $ cd cachedir $ proxypypi run [buildout] index = http://pypi-proxy-machine/simple/