Master & Minion: or the Dream of Python Automation

In this day and age, using hand-crafted servers is quickly becoming a losing proposition. Debugging a Python application that runs on custom made servers gets extremely tricky, and since apps no longer run on a single server, releases become increasingly resource intensive. It is therefore vital to have a process to control multiple servers simultaneously from a single command. This talk goes through real life and practical examples to roll out orchestration and configuration management projects using Python.

Transcript

1. MASTER & MINIONS MASTER & MINIONS OR THE DREAM OF PYTHON AUTOMATION OR THE DREAM OF PYTHON AUTOMATION CALVIN HENDRYX-PARKER, CTO CALVIN HENDRYX-PARKER, CTO SIX FEET UP SIX FEET UP
2. CERTAIN THINGS SHOULD BE HAND CERTAIN THINGS SHOULD BE HAND CRAFTED. CRAFTED.
3. BUT NOT YOUR INFRASTRUCTURE BUT NOT YOUR INFRASTRUCTURE
4. BEAUTIFUL UNIQUE SNOWFLAKES ARE BEAUTIFUL UNIQUE SNOWFLAKES ARE NOT REPRODUCIBLE. NOT REPRODUCIBLE.
5. PETS PETS
6. CATTLE CATTLE
7. RULES OF DEVOPS CLUB RULES OF DEVOPS CLUB » The rst rule of DevOps Club is: You do not log into servers » The second rule of DevOps Club is: You do not log into servers » Third rule of DevOps Club: If your deployment fails, rollback » Fourth rule: All artifacts will be stored in source control » Fifth rule: Only one deployment at a time » Sixth rule: No one offs, No special cases » Seventh rule: Deployments will go on as long as they have to » And the eighth and nal rule: If this is your rst night at DevOps Club, you have to push to prod.
8. TOOLS TOOLS To enter the DevOps world, you need to know what tools are available to you. Python has many great tools available to use such as SaltStack and the AWS Boto3 library.
9. FROM THE CLOSET TO THE CLOUD FROM THE CLOSET TO THE CLOUD
10. SINGLE SERVER MONOLITH SINGLE SERVER MONOLITH ┌────────────────────────────────────┐ │ Web/App/Database │ │ ┌───────────────────────────┐ │ │ │ Nginx │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ Varnish │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ HAProxy │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ Plone Instances x 4 │ │ │ └───────────────────────────┘ │ │ │ │ ┌───────────────────────────┐ │ │ │ ZEO Storage Server │ │ │ └───────────────────────────┘ │ │ │ └────────────────────────────────────┘
11. CLOUD OPTIMIZED CLOUD OPTIMIZED
12. AUTOMATION AUTOMATION
13. ENTER SALTSTACK AND BOTO3 ENTER SALTSTACK AND BOTO3
14. NETWORK END GOAL NETWORK END GOAL ┌─────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ control-vpc │ │ prod-vpc │ │ ┌───────────────────────────────────┐ │ │ ┌───────────────────────────┐ │ │ │ control-public-az1-subnet │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ prod-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌───────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ │ │ │ salt.esc.nd.edu │ │ │────Group ──────│ └────────────────────────────┘ │ │ │ │ │ │ │ Rules │ ┌────────────────────────────┐ │ │ │ │ Salt Master/VPN │ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────┘ │ │ │ │ prod-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────┘ │ │ └┬───────────────────────────┘ │ │ └─────────────────────────────────────────┘ │ └─────────────────────────────┘ │ │ └────────────────────────────────────┘ │ │ │ ┌────────────────────────────────────┐ │ │ test-vpc │ │ │ ┌───────────────────────────┐ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ test-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ └──────────────Group ─────────────────│ └────────────────────────────┘ │ Rules │ ┌────────────────────────────┐ │ │ │ ├─┐ │ │ │ │ │ │ │ │ test-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ └┬───────────────────────────┘ │ │ │ └─────────────────────────────┘ │ └────────────────────────────────────┘
15. BIT OF A CHICKEN AND EGG PROBLEM BIT OF A CHICKEN AND EGG PROBLEM # VPC c = boto.vpc.connect_to_region(region) vpc = c.create_vpc('172.20.0.0/16') vpc.add_tag('Name', 'control-vpc') c.modify_vpc_attribute(vpc.id,enable_dns_hostnames=True)
16. TOOL TIP TOOL TIP KEEP YOUR SECRETS… SECRET KEEP YOUR SECRETS… SECRET $ lpass login # LastPass $ $(lpass-env export 'AWS SFU Calvin') $ eval $(op signin sixfeetup) # 1Password $ awsenv "AWS SFU Calvin" $ env | grep AWS AWS_ACCESS_KEY_ID=DEADBEEFCAFE AWS_SECRET_ACCESS_KEY=123jasdfadsOof9akayo5peey0cow AWS_DEFAULT_REGION=us-east-1
17. BOOTSTRAP CONTINUED BOOTSTRAP CONTINUED # EC2 data_path = os.path.join(os.path.dirname(__file__), 'bootstrap-master.sh') with open(data_path) as script: bootstrap = script.read() master = ec2.run_instances( 'ami-55ef662f', subnet_id=subnet.id, security_group_ids=[group.id], key_name=ssh_key, instance_type='t2.micro', user_data=bootstrap ) master.instances[0].add_tag('Name', 'Salt Master')
18. WHAT IS SALTSTACK? WHAT IS SALTSTACK?
19. WHAT SETS SALT APART? WHAT SETS SALT APART? » Remote Execution » Event-Driven Orchestration » Agent or Agent-less Operation » Cloud Provisioning » Speed and Scalability
20. LONELY MINIONS LONELY MINIONS
21. MASTER AND MINIONS MASTER AND MINIONS
22. NOW WE ORCHESTRATE NOW WE ORCHESTRATE And build a test environment $ salt-run state.orchestrate orch.deploy-environment pillarenv=prod $ salt-run state.orchestrate orch.deploy-environment pillarenv=test
23. INFRASTRUCTURE (SANS HUMANS) INFRASTRUCTURE (SANS HUMANS) ┌─────────────────────────────────────────┐ ┌────────────────────────────────────┐ │ control-vpc │ │ prod-vpc │ │ ┌───────────────────────────────────┐ │ │ ┌───────────────────────────┐ │ │ │ control-public-az1-subnet │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ prod-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ ┌───────────────────┐ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ │ │ │ salt.esc.nd.edu │ │ │────Group ──────│ └────────────────────────────┘ │ │ │ │ │ │ │ Rules │ ┌────────────────────────────┐ │ │ │ │ Salt Master/VPN │ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────┘ │ │ │ │ prod-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ └───────────────────────────────────┘ │ │ └┬───────────────────────────┘ │ │ └─────────────────────────────────────────┘ │ └─────────────────────────────┘ │ │ └────────────────────────────────────┘ │ │ │ ┌────────────────────────────────────┐ │ │ test-vpc │ │ │ ┌───────────────────────────┐ │ │ │ │ ├─┐ │ │ │ │ │ │ │ │ │ │ test-public-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ Security │ └┬──────────────────────────┘ │ │ └──────────────Group ─────────────────│ └────────────────────────────┘ │ Rules │ ┌────────────────────────────┐ │ │ │ ├─┐ │ │ │ │ │ │ │ │ test-private-az1-subnet │ │ │ │ │ │ │ │ │ │ │ │ │ │ └┬───────────────────────────┘ │ │ │ └─────────────────────────────┘ │ └────────────────────────────────────┘
24. ORCHESTRATING NEW CODE RELEASES ORCHESTRATING NEW CODE RELEASES
25. ZERO DOWNTIME RELEASES ZERO DOWNTIME RELEASES {%- for app_server in app_servers %} ... # Stop Varnish, indirectly removing this app server from the NetScaler stop-{{ app_server }}-varnish: salt.function: - name: service.stop - arg: - {{ varnish.service }} - tgt: {{ app_server }} ...
26. PREPARE FOR THE RELEASE PREPARE FOR THE RELEASE ... # Stop Instances stop-{{ app_server }}-instances: salt.function: - name: supervisord.stop - arg: - all - tgt: {{ app_server }}
27. OPTIONAL AD-HOC CODE RELEASES OPTIONAL AD-HOC CODE RELEASES ... # Check out a specific revision {% set branch = salt['pillar.get']('plone:branch', 'dev') %} # to specify on the command-line: pillar='{"plone": {"branch": "f5d9859"}}' checkout-code-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: git.checkout - kwarg: user: webuser rev: {{ branch }} - arg: - {{ buildout_dir }} ...
28. HANDLE RELEASE TASKS HANDLE RELEASE TASKS # Run the buildout on the app server, but only run the generic setup on one server run-buildout-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: cmd.run - kwarg: cwd: {{ buildout_dir }} runas: webuser - arg: - {{ buildout_dir }}/env/bin/buildout -N
29. HANDLE POST RELEASE TASKS HANDLE POST RELEASE TASKS # Install Gulp bits install-gulp-{{ app_server }}: salt.function: - name: npm.install - tgt: {{ app_server }} - kwarg: runas: webuser dir: {{ theme_dir }} # Run the Gulp Build run-gulp-{{ app_server }}: salt.function: - tgt: {{ app_server }} - name: cmd.run - arg: - gulp build - kwarg:
30. NOT JUST FOR RELEASES NOT JUST FOR RELEASES » Migrating Production Data back to Testing » Scheduled tasks such as Database Backups
31. GRABBING PRODUCTION DATA GRABBING PRODUCTION DATA # Find the dev master db for use later {% set ns = namespace(dev_master=None) %} {% for dev_master in dev_db_servers if salt.saltutil.cmd(dev_master, 'file.search', arg=['/var/run/keepalive.state', 'MASTER']).get(dev_master).get('ret') %} {% if loop.first %} {% set ns.dev_master = dev_master %} {% endif %} {% endfor %} # Transfer Prod database {% for prod_slave in prod_db_servers if salt.saltutil.cmd(prod_slave, 'file.search', arg=['/var/run/keepalive.state', 'BACKUP']).get(prod_slave).get('ret') %} {% if loop.last %} sync prod database to dev:
32. SOUNDS TOO EASY SOUNDS TOO EASY The road was bumpy for sure. Satisfying the rules for no special cases was tricky.
33. MINDFULNESS MINDFULNESS There should be one, and preferably only one, obvious way to do it. Although that way may not be obvious at rst unless you’re Dutch. – Tim Peters, The Zen of Python
34. THE JOURNEY OPERATING SYSTEMS THE JOURNEY OPERATING SYSTEMS » 2a67758 Editing requirements to run properly on amazon linux » 472d844 Refactoring to run CentOS 7 machines » dd67b7a Refactoring for FreeBSD What happened here?
35. OTHER RECOMMENDATIONS OTHER RECOMMENDATIONS » Implement a CI strategy to test your infrastructure with tools like » Create or use tools to help you trace requests in the cloud Kitchen-Salt https://github.com/sixfeetup/aws-log-tools
36. LIVING THE DREAM LIVING THE DREAM
37. QUESTIONS? QUESTIONS? CALVIN@SIXFEETUP.COM CALVIN@SIXFEETUP.COM @calvinhp