Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Practicing Continuous Deployment - DISQUS - PyC...

Practicing Continuous Deployment - DISQUS - PyCon 2012

Continuous deployment (and testing) has started to become a reality for many companies. It brings to light one of the many problems that face large product teams, but also creates some of its own.

This talk will focus on the pros and cons of continuous deployment, how DISQUS switched from the recurring release cycle to continuous releases, as well as providing tips and arguments for adopting it in your workplace.

David Cramer

March 10, 2012
Tweet

More Decks by David Cramer

Other Decks in Technology

Transcript

  1. Continuous Deployment # Update the site every 5 minutes */5

    * * * * cd /www/example.com \ && git pull \ && service apache restart Saturday, March 10, 12
  2. When is it ready? - Reviewed by peers - Passes

    automated tests - Some level of QA Saturday, March 10, 12
  3. - Develop features incrementally - Release frequently - Smaller doses

    of QA - Culture Shock - Stability depends on test coverage - Initial time investment The Good The Bad Saturday, March 10, 12
  4. Development - Automate testing of complicated processes and architecture -

    Simple can be better than complete - Especially for local development - python setup.py {develop,test} - Puppet, Chef, Buildout, Fabric, etc. Saturday, March 10, 12
  5. Production Staging CI Server Macbook PostgreSQL Memcache Redis Solr Apache

    Nginx RabbitMQ PostgreSQL Memcache Redis Solr Apache Nginx RabbitMQ PostgreSQL Memcache Redis Solr Apache Nginx RabbitMQ PostgreSQL Apache Memcache Redis Solr Nginx RabbitMQ (and 100 other painful-to-configure services) Saturday, March 10, 12
  6. Bootstrapping Local - Simplify local setup - git clone dcramer@disqus:disqus.git

    - make - python manage.py runserver - Need to test dependancies? - virtualbox + vagrant up Saturday, March 10, 12
  7. Deploy features to portions of a user base at a

    time to ensure smooth, measurable releases https://github.com/disqus/gargoyle Saturday, March 10, 12
  8. from gargoyle import gargoyle def my_view(request): if gargoyle.is_active('awesome', request): return

    'new happy version :D' else: return 'old sad version :(' • Iterate quickly by hiding features • Early adopters are free QA Saturday, March 10, 12
  9. SWITCHES = { # enable my_feature for 50% 'my_feature': range(0,

    50), } def is_active(switch): try: pct_range = SWITCHES[switch] except KeyError: return False ip_hash = sum([int(x) for x in ip_address.split('.')]) return (ip_hash % 100 in pct_range) Saturday, March 10, 12
  10. Integration Requirements - Developers must know when they’ve broken something

    - IRC, Email, IM - Support proper reporting - XUnit, Pylint, Coverage.py - Painless setup - apt-get install jenkins * https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+on+Ubuntu Saturday, March 10, 12
  11. Shortcomings - False positives - Reporting isn't accurate - Services

    fail - Bad Tests - Test coverage - Regressions on untested code - Feedback delay - Integration tests vs Unit tests Saturday, March 10, 12
  12. Fixing False Positives - Re-run tests several times on a

    failure - Report continually failing tests - Replace external service tests with a functional test suite Saturday, March 10, 12
  13. Maintaining Coverage - Raise awareness with reporting - Fail/alert when

    coverage drops on a build - Commit tests with code - Coverage against commit di for untested regressions - Utilize code review Saturday, March 10, 12
  14. Speeding Up Tests - Write true unit tests - vs

    slower integration tests - Mock external services - Distributed and parallel testing - Matrix builds Saturday, March 10, 12
  15. <You> Why is mongodb-1 down? <Ops> It’s down? Must have

    crashed again Saturday, March 10, 12
  16. Meaningful Metrics - Rate of tra c (not just hits!)

    - Business vs system - Response time (database, web) - Exceptions - Social media - Twitter Saturday, March 10, 12
  17. Getting Started - Package your app - Value code review

    - Ease deployment; fast rollbacks - Setup automated tests - Gather some easy metrics Saturday, March 10, 12
  18. Going Further - Build an immune system - Automate deploys,

    rollbacks (maybe) - Adjust to your culture - There is no “right way” - SOA == great success Saturday, March 10, 12
  19. References - Gargoyle (feature switches) https://github.com/disqus/gargoyle - Sentry (log aggregation)

    https://github.com/dcramer/sentry - Jenkins CI (continuous integration) http://jenkins-ci.org/ - Phabricator (code reviews, bug tracking) https://phabricator.org - Graphite (metrics) http://graphite.wikidot.com/ code.disqus.com Saturday, March 10, 12