Developing & Deploying "Large" Scale Web Applications

June 12th, 2012 @ Urban Airship

Transcript

1. Developing “large” Scale Web Applications Matt Robenolt // github.com/mattrobenolt OR @mattrobenolt s/developing/deploying/
2. Hi.
3. What this is not
4. + What this is not This is not a definitive guide to solving for x. This is not a solution to your problems. All problems are unique and must be approached pragmatically.
5. What this is
6. + What this is This is an exploration of potentially new tools. This is an introduction to new ideas and concepts. This is the start of a discussion. This is what worked for us.
7. What do I mean by “large”?
8. + What do I mean by “Large”? Lots and lots of Moving pieces Django, Node.js, Redis, SSL, virtual hosts, MongoDB, CDN, etc.
9. + What do I mean by “Large”? Life Beyond python manage.py runserver http://localhost:8000/
10. The Story of Drund
11. + Story of Drund Standard, out-of-the-box Django project First ~6 months
12. + Story of Drund Things were... GREAT!
13. + Story of Drund Developing locally ./manage.py runserver
14. + Story of Drund Deploying to production ssh matt@… -- svn up; touch /tmp/reload;
15. + Story of Drund Our stack Today
16. + Story of Drund Our stack Today Django, MySQL, Redis, Node.js, MongoDB, RabbitMQ, CDN, SSL-only, LESS, Require.js, Hogan.js, background worker, etc.
17. + Story of Drund Development is a bitch.
18. + Story of Drund Deploying is a bitch.
19. What’s the problem?
20. + What’s the problem? Django Nothing wrong with Django per se. Conditioned us for no external dependencies. Makes getting up and started super simple. SQLite3, Dummy cache, local memory cache, etc.
21. + What’s the problem? Node.js An external process, so needs another port. Requires two Terminal windows to spawn Django + Node.js app. Cookies aren’t shared across different ports, must match domain:port combo.
22. + What’s the problem? Redis, MongoDB, RabbitMQ Can’t be mocked in Django realistically like MySQL <-> SQLite. Annoying to run on local machine, mismatched configuration/versions, blah blah.
23. + What’s the problem? Less, Require.js, Hogan.js Assets must be compiled before being loaded into the browser. Tedious tasks during rapid development.
24. + What’s the problem? CDN …wut? Introduces occasional cross-domain issues. File uploads.
25. + What’s the problem? SSL Django can’t handle SSL. Real web server required.
26. + What’s the problem? Virtual hosts /etc/hosts becomes unwieldy. No wildcards.
27. + What’s the problem? 127.0.0.1 drund.local 127.0.0.1 www.drund.local 127.0.0.1 api.drund.local 127.0.0.1 admin.drund.local 127.0.0.1 team.drund.local 127.0.0.1 partners.drund.local 127.0.0.1 matt.drund.local 127.0.0.1 nick.drund.local 127.0.0.1 kevin.drund.local 127.0.0.1 mike.drund.local 127.0.0.1 adam.drund.local 127.0.0.1 lee.drund.local
28. + What’s the problem? Configuration Management Django uses settings.py. Node.js uses settings.json. Both represent very similar data. Database, cache, etc.
29. + What’s the problem? Deployments git pull with Fabric. Every server in the cluster has different definitions of “update”. System dependencies, libxml, mysql-client, etc.
30. Re-architect ALL THE THINGS!
31. Virtual Hosts
32. + Virtual Hosts Forget about /etc/hosts. You can point a real DNS entry to 127.0.0.1! Real DNS gives you wildcard and CNAME entries.
33. + Virtual Hosts local.drund.net. A 127.0.0.1 *.local.drund.net. CNAME local.drund.net See Also: http://xip.io
34. Make & The Makefile
35. + Make & The Makefile Not just for “Real” compiling. It’s pretty much just /bin/sh. Simplify long or tedious commands. You’ll use it for a lot of things, I promise.
36. + Make & The Makefile Constructed of “Targets”. Targets can be real or not real. 98%* of your targets will be “PHONY”. Targets are essentially commands executed through `make`. make <target> * Not a real statistic.
37. + Make & The Makefile Targets support dependencies. “static depends on css and js first” targets == functions or commands static: @rm -rf ./static ./manage.py collectstatic --noinput -v0 clean: find . -name "*.pyc" -delete
38. + Make & The Makefile How we compile our static assets before:
39. + Make & The Makefile ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:
40. + Make & The Makefile FFuuuuuu!!! ../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > ./media/main/css/kovu.css ../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} $(find ./media -name *.mustache) ../tools/r.js/dist/r.js -o kovu.build.js ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 How we compile our static assets before:
41. + Make & The Makefile How we compile our assets with make:
42. + Make & The Makefile make static How we compile our assets with make:
43. + Make & The Makefile make static How we compile our assets with make:
44. all: css js templates static css: @../tools/less.js/bin/lessc --yui-compress -O2 ./media/main/css/kovu.less > \ ./media/main/css/kovu.css @echo "css compiled." js: templates @node ../tools/r.js/dist/r.js -o kovu.build.js @echo "js compiled <kovu.build.js>" templates: @../tools/hogan.js/bin/hulk --wrapper amd --outputdir {{current_dir}} \ $(shell find ./media -name *.mustache) @echo "Templates smushed successfully." static: css js templates @rm -rf ./static ./manage.py collectstatic --noinput --ignore=*.mustache --ignore=*.less -v0 + Make & The Makefile Makefile
45. + Make & The Makefile Why not Fabric? Fabric is amazing for Python projects. We use it. Not the best for non-python. Makefiles are more generic. Choose what works for you, where and when it makes sense. * bitprophet: I do love Fabric. ;)
46. m4
47. + M4 Macro processor. Very “Dumb”. Basic string replacements. Small macros. Inheritance. Allows us to build configuration files for both Django and Node.js without duplication. Allows us to build configuration files for different environments.
48. + M4 C-ish define statements. define(`__NAME__', `Matt') Hello __NAME__
49. + M4 Odd syntax. Surprisingly does a lot.
50. + m4 define(`__DEBUG__', `True') define(`__DATABASE_ENGINE__', `mysql') define(`__DATABASE_NAME__', `drund') define(`__DATABASE_USER__', `root') define(`__DATABASE_PASSWORD__', `root') define(`__DATABASE_HOST__', `127.0.0.1') define(`__DATABASE_PORT__', `3306') m4/base.m4 define(`__DEBUG__', `False') define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `$uper$ecret!') define(`__DATABASE_HOST__', `10.0.0.0') m4/production.m4 define(`__DATABASE_USER__', `drund') define(`__DATABASE_PASSWORD__', `heywereonstaging!') m4/staging.m4
51. + m4 DEBUG = __DEBUG__ DATABASES = { 'default': { 'ENGINE': 'django.db.backends.__DATABASE_ENGINE__', 'NAME': '__DATABASE_NAME__', 'USER': '__DATABASE_USER__', 'PASSWORD': '__DATABASE_PASSWORD__', 'HOST': '__DATABASE_HOST__', 'PORT': __DATABASE_PORT__', } } m4/settings.py.m4 { "database": { "name": "__DATABASE_NAME__", "user": "__DATABASE_USER__", "password": "__DATABASE_PASSWORD__", "host": "__DATABASE_HOST__", "port": __DATABASE_PORT__ } } m4/settings.json.m4
52. + m4 m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json settings: m4 m4/base.m4 m4/production.m4 m4/settings.py.m4 > settings.py m4 m4/base.m4 m4/production.m4 m4/settings.json.m4 > settings.json Makefile Build it!
53. Vagrant http://vagrantup.com
54. + Vagrant “Virtualized development made easily.” Headless, local virtual machine. port forwarding. Easy bootstrapping with Chef, Puppet, or Shell script. vagrant up Runs our “Utilities” without cluttering local machine. (MySQL, Redis, RabbitMQ, MongoDB)
55. + Vagrant Vagrant::Config.run do |config| config.vm.box = "precise64" config.vm.box_url = "http://files.vagrantup.com/precise64.box" config.vm.forward_port 3306, 3306 # MySQL config.vm.forward_port 6379, 6379 # Redis config.vm.forward_port 5672, 5672 # RabbitMQ config.vm.forward_port 27017, 27017 # MongoDB config.vm.provision :shell, :path => "bootstrap.sh" end Vagrantfile
56. + Vagrant Downloads “box” for first use. Creates Virtual Machine using VirtualBox. Forwards necessary ports, so I can access via 127.0.0.1:<port> Executes bootstrap.sh after build. vagrant ssh # SSH into the server vagrant halt # Shut down gracefully vagrant destroy # Delete the VM entirely
57. + Vagrant #!/bin/sh # First, let's check if everything is installed before we go installing shit again if [ -f /usr/sbin/mysqld ]; then \ exit 0 fi # MySQL root password echo mysql-server-5.1 mysql-server/root_password password root | debconf-set-selections echo mysql-server-5.1 mysql-server/root_password_again password root | debconf-set-selections apt-get update && apt-get install mysql-server -y echo "grant all privileges on *.* to 'root'@'%' identified by 'root' with grant option;"| mysql -u root -proot echo "Creating \"drund\" database..." echo "create database drund;" | mysql -u root -proot echo "Binding MySQL to 0.0.0.0..." cp /etc/mysql/my.cnf /etc/mysql/my.cnf.old sed -e's/127.0.0.1/0.0.0.0/' /etc/mysql/my.cnf.old > /etc/mysql/my.cnf echo "Restarting MySQL..." service mysql restart exit 0 bootstrap.sh
58. + Vagrant vm: vagrant up ./manage.py syncdb --migrate --noinput Makefile
59. Almost... but not quite good enough.
60. + Almost... but not quite good enough. # Only run once make settings make vm # Get the server running make static ./manage.py runserver # Listening @ http://www.local.drund.net:8000/ Current status
61. + Almost... but not quite good enough. What’s missing? make static needs to be run every time a file is changed. Node.js is still running on another port. No SSL. No way to simulate a CDN.
62. Node-HTTP-Proxy & “Node-testserver” https://github.com/nodejitsu/node-http-proxy
63. + Node-http-proxy & “node-testserver” Node.js... wut? Stand in as a dummy/fancier nginx. Proxy virtual host requests to different apps/ports. Pipe stdout/stderr like expected to console. Render Less and Mustache templates on the fly. “CDN” is just another virtual host. SSL.
64. + Node-http-proxy & “node-testserver” in a nutshell http-proxy spawns server on 443, terminates SSL, and processes vhosts as expected. node0.local.drund.net -> localhost:9001 # A miscellaneous Node.js app *.local.drund.net -> localhost:8000 # Main Django app testcdn.local.azork.com -> localhost:9002 # Internal Express media server
65. + Node-http-proxy & “node-testserver” in a nutshell Requests to internal media server are resolved through Django’s findstatic command. ./manage.py findstatic --first <path> LESS files are compiled into CSS when requested. Mustache templates are rendered on the fly. main.css -> main.less hello.template.js -> hello.template.mustache
66. + Node-http-proxy & “node-testserver” Added Bonus! Django is no longer serving up static media. Load assets concurrently. It actually makes a difference.
67. sudo make testserver https://www.local.drund.net/
68. Jenkins http://jenkins-ci.org/
69. + Jenkins Continuous Integration & Continuous Deployment Nirvana
70. + Jenkins Post-commit hooks to run builds automatically. Executes a series of Shell commands. Builds new virtualenv for each build. Runs tests. Fail if tests fail. Notify team. Build deb package for source code. Push package to private apt repository. apt-get install drund-{component}-{branch} -y on servers with Fabric. Integrates nicely with IRC.
71. Deb packaging
72. + Deb packaging Why a system package? Originally skeptical myself. Very easily manage dependencies outside of Python/pip. Post-install hooks. No compiling on production servers.
73. + Deb packaging Drund is broken up into many packages by code branch. Packages can be mixed and matched depending on roles. drund-src-master # Main Drund source code. Only. drund-www-master # Depends on drund-src-master, nginx config drund-worker-master # Depends on drund-src-master, starts background worker drund-activity-master # Node.js app drund-proxy-master # Node.js app drund-cron-master # Depends on drund-src-master, cron.d files.
74. + Deb packaging drund-src-master Virtual package for drund-src-r{build}. Essentially: drund-srcdepends on drund-src-r{build} drund-src-r100 and drund-src-r101 can be installed at the same time. post install manages a symlink. 100% new environment for each build. Easy to roll back. Just repoint symlink. No processes are spawned.
75. + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer: Drund <ops@drund.com> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:
76. + Deb packaging Source: drund-src-master-r100 Section: web Priority: optional Maintainer: Drund <ops@drund.com> Build-Depends: debhelper (>= 5) Standards-Version: 3.8.4 Homepage: https://www.drund.com Package: drund-src-kovu Architecture: all Depends: drund-src-master-r100 Package: drund-src-master-r100 Architecture: all Depends: lsb-base (>= 3.0-6), python (>= 2.7), mysql-client (>= 5.1), ... Drund-src control file:
77. + Deb packaging Install via Apt-get apt-get install drund-src-master
78. Continuous Integration/deployments
79. + Continuous Integration/Deployments Tests are run automatically before anything is deployed. Tests are run after every single push. It’s actually impossible to deploy code with broken tests. Jenkins can be configured to run a Fabric command to update after a successful build. It’s just another shell command. Annoying at first. You will make mistakes.
80. + Continuous Integration/Deployments Forced to write better code. Forced to write more tests. Once your environment is configured, everything is done correctly. Every time. Computers are pretty good at that. Easy to bring up staging or development servers.
81. New & improved workflow
82. + New & Improved Workflow master branch is 100% as stable as possible. Production only. Code is only merged in from develop. develop contains code ready to be merged into master at any time. Continuously deployed to dev/staging servers. Feature branches off of develop. If bad code is pushed to develop, Jenkins complains and posts in IRC. If all is good, code review, queue up for merge into master.
83. + New & Improved Workflow Hit the ground running in very little time. git clone git@... make init # Grabs submodules, create virtualenv, installs requirements
84. + New & Improved Workflow Run the project. sudo make testserver # sudo is needed for port 443.
85. + New & Improved Workflow Deploy code to staging. git checkout develop git merge feature-branch # Or Pull Request git push origin develop
86. + New & Improved Workflow Deploy to production. git checkout master git merge develop # Preferably a Pull Request git push origin master
88. Thanks. @Mattrobenolt github.com/mattrobenolt mattrobenolt.com