Juergen Brendel: Deploying test and production systems with Ansible

= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
Juergen Brendel:
Deploying test and production systems with Ansible
= = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = = =
@ Kiwi PyCon 2014 - Sunday, 14 Sep 2014 - Track 1
http://kiwi.pycon.org/

**Audience level**

Intermediate

**Description**

This talk presents Ansible as a light-weight, simple and effective configuration management tool. We will see why configuration management is important and how we can use Ansible to automatically deploy and test full production clusters - showcased with a sample Django application - with just a single command and in a number of different environments.

**Abstract**

With modern configuration, deployment and orchestration systems, software developers can easily maintain, bring up and tear down local test systems, staging, demonstration and production systems, which are 100% identical to each other.

This presentation will give an introduction to Ansible, a modern, simple and efficient configuration management system. We will see how we can deploy complete clusters (with load balancers, database and some Django application servers), reliably and fully automated, either on dedicated servers, on local virtual machines or in the cloud.

If all goes well, there will even be a live demo in which we will bring up a complete cluster or two in different environments, to illustrate the power of this approach.

**YouTube**

https://www.youtube.com/watch?v=Dg5TSfLHW3Q

Transcript

1. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Automated deployment with Ansible Juergen Brendel juergen@brendel.com @BrendelConsult
2. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Summary • Configuration management background • Ansible intro • Unified development, test and deployment environments • Ansible and the cloud • Contributing to Ansible
3. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Configuration Management: Why and how
4. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult What configuration? • System packages • Settings • Users/groups • Applications
5. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Arcane magic Configuring servers How do you configure a server? Manual instructions Scripts CM tools Automation!
6. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ensure all system packages are updated. CM Tools Describe the desired state Ensure that user “xyz” exists. Ensure package “apache” is installed. Have latest sources: Clone repo, update if it exists already. Ensure package “postgres” v9.1 is installed. Ensure DB “app_data” exists with password “****”.
7. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult CM Tools variety • Puppet (2005) • Chef (2009) • Salt (2011) • Ansible (2012) • Fabric • Scripts “powerful, feature-rich, enterprisy” “simple, fast, good for most things” “not really CM tools”
8. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible: Intro and key concepts
9. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible overview • “Orchestration engine” for CM and deployment • Written in Python • Uses YAML • “Playbooks” • Config specs or explicit commands • Linux and Windows
10. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible simplicity Key points: – No central configuration server – No key management – No agent to install on target machine – Explicit order Requirements: – Need SSH access (with key or password) – Need Python installed on target machine
11. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible architecture Server 1 Server 2 Server 3 Server 4 Server 5 Your laptop
12. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Modules Hundreds of them. They know how to do stuff... - Command - Shell - Script - Copy - Sync - Templates - Line ops - Install packages - Users and groups - Networking - Services - Repositories - Message queues - Monitoring - Notification - Web servers - Database servers - Cloud infrastructure
13. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult How does it work? Server Your laptop Python module: “install apache” Run module Delete module on server
14. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Inventory and groups Define hosts, organized in groups • by function • by location • by hosting provider • ...
15. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Inventory and groups Define hosts, organized in groups [europe] server1.somehoster.co.uk server2.otherhoster.de [north­america] host­a.serverhost.com host­b.serverhost.com [frontend] server1.somehoster.co.uk host­b.serverhost.com [backend] server2.otherhoster.de host­a.serverhost.com
16. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Adhoc commands Single commands, applied to groups $ ansible ­i hosts europe ­a “uname ­a” $ ansible ­i hosts frontend ­a “/sbin/reboot” ­f 3
17. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Playbooks ­­­ ­ hosts: frontend sudo: yes tasks: ­ name: Update the system apt: pkg=nginx state=latest ­ name: Create the user account user: name=appuser shell=/bin/bash state=present ­ name: Copy files to remote user's home copy: > src=files/names.txt dst=/home/appuser owner=appuser mode=0644
18. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Variables ­­­ ­ hosts: all sudo: yes vars: username: appuser tasks: ­ name: Create the user account user: > name={{ username }} shell=/bin/bash state=present
19. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Project layout 1 / my_hosts group_vars/ all frontend backend europe north­america site.yml
20. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Project layout 2 / ansible.cfg deploy_hosts staging_hosts group_vars/ all frontend backend europe north­america host_vars/ server1.somehoster.co.uk host­b.serverhost.com site.yml roles/ common/ tasks/ main.yml handlers/ main.yml templates/ sshd_config.j2 files/ my_script.sh vars/ main.yml web/ ... db/ ...
21. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Playbooks with roles ­­­ ­ hosts: frontend sudo: yes roles: ­ common ­ web
22. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Local environments
23. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Dev, Test, Deploy Common issues
24. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Dev, Test, Deploy Why didn't we catch this bug in testing? How do I setup my development environment? We can't reproduce the issues. I don't have access to our test server. “It works for me...” (shrug)
25. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Dev, Test, Deploy Why didn't we catch this bug in testing? How do I setup my development environment? We can't reproduce the issues. I don't have access to our test server. “It works for me...” (shrug) Wouldn't this be nice instead? Single command: Dev environment created Single command: Test environment created
26. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Vagrant • Use Vagrant to spin-up VMs • local (VirtualBox, VMware, etc.) • cloud (EC2) • Use Ansible as 'provisioner' • Make an inventory file with just your VM • Point at same playbook as before
27. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible and “the cloud”
28. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult The rise and rise of APIs APIs Local Infrastructure Services This is really cool!
29. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Ansible 'cloud' modules Public cloud • Amazon AWS • Google Compute • Azure • Digital Ocean • Rackspace • Linode Public cloud • Amazon AWS • Google Compute • Azure • Digital Ocean • Rackspace • Linode Private cloud • OpenStack • Eucalyptus • Vsphere • Docker • libvirt Private cloud • OpenStack • Eucalyptus • Vsphere • Docker • libvirt
30. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Example: AWS modules EC2 / infrastructure • Instances • Images • VPCs • Load balancers Services • S3 • Route 53 • Databases • Cache
31. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Example: Create instances # My hostfile: Empty groups [frontend] [backend] [my­server­group]
32. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Example: Create instances ­ local_action: module: ec2 key_name: my­key group: my­security­group instance_type: t2.micro image: ami­120abc90 region: us­east­1 count: 3 register: ec2results
33. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Example: Create instances ­ local_action: module: ec2 key_name: my­key group: my­security­group instance_type: t2.micro image: ami­120abc90 region: us­east­1 count: 3 register: ec2results { "ec2results": { "changed": true, "instance_ids": [ "i­da9fbbe4", ... ], "instances": [ { "id": "i­da9fbbe4", "instance_type": "t1.micro", "image_id": "ami­0729b73d", "region": "us­east­1", "public_ip": "54.253.65.41", "private_ip": "10.248.39.121", ... }, ...
34. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Example: Create instances ­ local_action: module: ec2 key_name: my­key group: my­security­group instance_type: t2.micro image: ami­120abc90 region: us­east­1 count: 3 register: ec2results ­ local_action: module: add_host hostname: {{ item.public_ip }} groupname: my­server­group with_items: ec2results.instances
35. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Contributing to Ansible
36. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Contributing • The perfect FOSS project? • Lots of small, independent modules • Written in Python For yourself: Any language, even bash. For contributions: Python preferred
37. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Getting started $ git clone git@github.com:ansible/ansible.git $ source ansible/hacking/env­setup … edit your module … $ ansible/hacking/test­module ­m ./my_module
38. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Sample module #!/usr/bin/python import datetime import json date = str(datetime.datetime.now()) print json.dumps({ “changed” : True, "time" : date }) Catch exceptions All output to stdout JSON
39. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Sample module #!/usr/bin/python DOCUMENTATION = ''' ­­­ module: filewrite short_description: write data to a file description: ­ Write data to start of file. Create if necessary. options: path: description: ­ The full path of the file/object to write to required: true default: null data: description: ­ The data to be written required: false default: some test data '''
40. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Sample module import os from ansible.module_utils.basic import * def main(): module = AnsibleModule( argument_spec = dict( path = dict(required=True), data = dict(default='some test', type='str'), ), ) path = module.params.get('path') data = module.params.get('data') try: if os.path.isfile(path): with open(path, "r") as f: d = f.read(len(data)) if d == data: module.exit_json(changed=False, other_data="Not changed.") with open(path, "w") as f: f.write(data) module.exit_json(changed=True, other_data="Wrote %s characters." % len(data)) except Exception as e: module.fail_json(msg=str(e)) main()
41. Automated Deployment with Ansible juergen@brendel.com @BrendelConsult Thank you very much! Questions: juergen@brendel.com Ansible docs: http://docs.ansible.com/ Ansible source: https://github.com/ansible/ansible Vagrant: http://www.vagrantup.com/ Example project: https://github.com/jbrendel/ansible_cluster_setup