Continuous Integration for Infrastructure

Lots of examples of emerging infrastructure testing patterns. Test Kitchen, Puppet, Docker, Serverspec, Packer, Bats and more. Presented at #craftconf in Budapest

1. Continuous Integration for Infrastructure Craft Conference, Budapest, 2014 Gareth Rushgrove
2. Who (Who is this person?)
3. @garethr
4. UK Government Digital Service
7. What is Continuous Integration (And what has that to do with infrastructure)
8. A common software development practice Gareth Rushgrove
9. Test every commit Gareth Rushgrove
10. Get fast feedback on changes Gareth Rushgrove
11. Discover bugs early, when they are easier to fix Gareth Rushgrove
12. Avoid working in isolation, encourage good team work Gareth Rushgrove
13. Lots of good tools to start with Gareth Rushgrove
14. Gareth Rushgrove jenkins-ci.org
15. Gareth Rushgrove jetbrains.com/teamcity
16. Gareth Rushgrove travis-ci.com
17. Gareth Rushgrove wercker.com
18. And lots more Gareth Rushgrove
19. Infrastructure as code means your infrastructure is software Gareth Rushgrove
20. So let’s apply traditional software engineering practices to infrastructure Gareth Rushgrove
21. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove
22. Configuration management! Acceptance testing! Verifying machine images ! Load testing! Provisioning Gareth Rushgrove
23. Configuration management! Acceptance testing! Verifying machine images ! Load testing! Provisioning Gareth Rushgrove
24. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove
25. Configuration management! Acceptance testing! Verifying machine images! Load testing! Provisioning Gareth Rushgrove
26. Infrastructure testing as code (A quick wrecker example)
27. Define your continuous integration setup in code too Gareth Rushgrove
28. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code: sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove
29. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code: sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove 1 2 3 4
30. View builds in a web browser Gareth Rushgrove
31. Gareth Rushgrove
32. - install-packages: packages: ruby1.9.1-dev - script: name: install bundler code: sudo gem install bundler - bundle-install - script: name: print ruby version code: ruby —version wercker.yaml Gareth Rushgrove
33. Gareth Rushgrove
34. You can do similar things with Jenkins Job Builder and with Travis Gareth Rushgrove
35. Testing config management code (The easy bit)
36. Gareth Rushgrove
37. Both Chef and Puppet have good testing support Gareth Rushgrove
38. Write unit tests for modules or cookbooks Gareth Rushgrove
39. rspec-puppet or chefspec Gareth Rushgrove
40. class sample { file { '/tmp/sample': ensure => present, } } sample.pp Gareth Rushgrove
41. describe "sample" do it { should create_file('/tmp/sample') } end sample_spec.rb Gareth Rushgrove
42. sample should contain File[/tmp/sample] ! Finished in 0.3881 seconds 1 example, 0 failures Gareth Rushgrove
43. Linting, code coverage, syntax checking Gareth Rushgrove
44. Lots of good material available on this subject Gareth Rushgrove
45. Gareth Rushgrove
46. Gareth Rushgrove
47. Acceptance testing infrastructure code (Testing with real machines)
48. You don’t need to be using config management tools to test your code Gareth Rushgrove
49. Run code against short lived virtual machines and make assertions against the resulting state Gareth Rushgrove
50. Gareth Rushgrove Test Kitchen
51. First the code we want to test Gareth Rushgrove
52. #!/bin/bash apt-get install ntp -y scripts/bootstrap.sh Gareth Rushgrove
53. Then a simple test kitchen configuration Gareth Rushgrove
54. --- driver: name: digitalocean ! provisioner: name: shell ! platforms: - name: ubuntu-12.04 ! suites: - name: default provisioner: script: scripts/bootstrap.sh .kitchen.yml Gareth Rushgrove
55. You can provision machines anywhere with different drivers Gareth Rushgrove
56. kitchen driver discover Gareth Rushgrove
57. Gareth Rushgrove
58. And have different platforms running in parallel Gareth Rushgrove
59. platforms: - name: ubuntu-14.04 - name: ubuntu-13.10 - name: centos-6.5 - name: centos-6.4 - name: ubuntu-12.04 - name: centos-6.4 - name: centos-6.4 - name: archlinux-2013.05 .kitchen.yml Gareth Rushgrove
60. And then write tests against the provisioned machines Gareth Rushgrove
61. Gareth Rushgrove serverspec.org
62. RSpec tests for your servers configured ! by Puppet, Chef or anything else Gareth Rushgrove
63. describe package('ntp') do it { should be_installed } end serverspec/sample_spec.rb Gareth Rushgrove
64. describe service('ntp') do it { should be_enabled } it { should be_running } end serverspec/sample_spec.rb Gareth Rushgrove
65. kitchen test -c Gareth Rushgrove
66. Package "ntp" should be installed ! Service "ntp" should be enabled should be running ! Finished in 0.11191 seconds 3 examples, 0 failures Finished verifying <default-ubuntu-1310> (0m7.39s). -----> Destroying <default-ubuntu-1310>... Digital Ocean instance <1493500> destroyed. Finished destroying <default-ubuntu-1310> (0m4.47s). Finished testing <default-ubuntu-1310> (3m59.40s). -----> Kitchen is finished. (3m59.95s) Gareth Rushgrove
67. Kitchen is finished. (3m59.95s) Gareth Rushgrove
68. Gareth Rushgrove garethr/do-test-kitchen
69. Gareth Rushgrove
70. Verifying machine images (AMIs and other artefacts)
71. Your virtual machine images need a pipeline too Gareth Rushgrove
72. We’ll start with an example using AMIs on Gareth Rushgrove
73. Gareth Rushgrove packer.io
74. Packer is a tool for creating identical machine images for multiple platforms from a single source configuration Gareth Rushgrove
75. Configuration is lots of JSON Gareth Rushgrove
76. { "variables": { "aws_access_key": "{{env `AWS_ACCESS_KEY_ID`}}", "aws_secret_key": "{{env `AWS_SECRET_ACCESS_KEY`}}" }, "builders": [{ "type": "amazon-ebs", "access_key": "{{user `aws_access_key`}}", "secret_key": "{{user `aws_secret_key`}}", "region": "eu-west-1", "source_ami": "ami-480bea3f", "instance_type": "t1.micro", "ssh_username": "ubuntu", "ami_name": "packer-serverspec-example-{{timestamp}}" }], "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "file", template.json Gareth Rushgrove
77. First build an AMI Gareth Rushgrove
78. Then install some packages and related configuration Gareth Rushgrove
79. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
80. First we’ll install puppet Gareth Rushgrove
81. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
82. #!/bin/bash ! # Install language locale as without can # interfere with package installation sudo apt-get install language-pack-en -y ! # Install Ruby sudo apt-get update sudo apt-get install ruby1.9.1 -y ! # Install puppet/facter sudo gem install puppet facter --no-ri --no-rdoc scripts/puppet.sh Gareth Rushgrove
83. Then run puppet to configure the machine Gareth Rushgrove
84. ! "provisioners": [{ "type": "shell", "script": "scripts/puppet.sh" }, { "type": "puppet-masterless", "manifest_file": "manifests/site.pp", "hiera_config_path": "hiera.yaml", "module_paths": ["modules"] } template.json Gareth Rushgrove
85. Packer supports a range of provisioners Gareth Rushgrove
86. Shell, File upload, Puppet, Chef, Salt, Ansible Gareth Rushgrove
87. How to test our built-but- not-saved image? Gareth Rushgrove
88. Run serverspec tests from within the new image Gareth Rushgrove
89. Only if the tests pass do we save the AMI Gareth Rushgrove
90. Gareth Rushgrove garethr/packer-serverspec-example
91. Gareth Rushgrove
92. Linux containers are another good artefact example Gareth Rushgrove
93. Testing! based on changes to the Dockerfile works in the same way Gareth Rushgrove
94. Gareth Rushgrove garethr/docker-spec-example
95. Gareth Rushgrove
96. Continuous load testing (The power of aggregate assertions)
97. Load testing is often! ad hoc Gareth Rushgrove
98. Load testing is more than just an application concern Gareth Rushgrove
99. Lots of open source tools Gareth Rushgrove
100. JMeter, Funkload, Tsung, Grinder, HTTPerf, Siege, Gatling, Locust, Vegeta, Gor … Gareth Rushgrove
101. Nothing is perfect for our purposes Gareth Rushgrove
102. Gareth Rushgrove We’ll use Gatling as an example
103. class ExampleSimulation extends Simulation { val httpConf = http.baseURL("http://www.example.com") ! val sample = scenario("Simple example").exec( http("Homepage").get("/").check(status.is(200))) ! setUp(sample.inject(ramp(3 users) over (10 seconds))) .protocols(httpConf) } ExampleSimulation.scala Gareth Rushgrove
104. Aggregate assertions are the killer feature of Gatling Gareth Rushgrove
105. .assertions( global.responseTime.mean.lessThan(750), ) ExampleSimulation.scala Gareth Rushgrove
106. .assertions( global.successfulRequests.percent.is(100), global.responseTime.max.lessThan(1000), global.responseTime.mean.lessThan(750), global.responseTime.percentile1.lessThan(800), global.requestsPerSec.greaterThan(10) ) ExampleSimulation.scala Gareth Rushgrove
107. Continuous load testing supplements rather than replaces ad hoc load testing Gareth Rushgrove
108. Gareth Rushgrove garethr/gatling-demo
109. Gareth Rushgrove
110. Entire infrastructures as code (Testing out provisioning)
111. So far we’ve tested units of infrastructure Gareth Rushgrove
112. This next example tests an entire system Gareth Rushgrove
113. As an example we’ll use Ansible Gareth Rushgrove
114. Gareth Rushgrove ansible.com
115. Specifically we’ll use the DigitalOcean module Gareth Rushgrove
116. First we define our system in code Gareth Rushgrove
117. --- droplets: - name: production.web.1 - name: production.web.2 - name: production.app.1 size: 64 - name: production.app.2 size: 64 - name: production.db.1 size: 69 vars/droplets.yml Gareth Rushgrove
118. For testing we’ll use a combination of serverspec and bats Gareth Rushgrove
119. Gareth Rushgrove sstephenson/bats
120. bats is great for testing combinations of shell scripts Gareth Rushgrove
121. #!/usr/bin/env bats ! @test "addition using bc" { result="$(echo 2+2 | bc)" [ "$result" -eq 4 ] } ! @test "addition using dc" { result="$(echo 2 2+p | dc)" [ "$result" -eq 4 ] } example.bats Gareth Rushgrove
122. Run out provisioning code inside a bats test Gareth Rushgrove
123. @test "no droplets exist" { run tugboat droplets [ "$status" -eq 0 ] [ "${lines[0]}" = "You don't appear to have any droplets." ] } ! @test "provisioner runs successfully" { run ansible-playbook -i hosts provision_digital_ocean.yml [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove
124. Confirm we have started some machines Gareth Rushgrove
125. @test “we can count the correct number of machines" { run bash -c "ansible -i hosts all --list-hosts | wc - l | tr -d ' '" [ "$status" -eq 0 ] [ "$output" -eq 5 ] } acceptance.bats Gareth Rushgrove
126. Serverspec has a mode of operation where it runs against remote hosts Gareth Rushgrove
127. @test "per host serverspec tests pass" { run rake spec [ "$status" -eq 0 ] } acceptance.bats Gareth Rushgrove
128. bats acceptance.bats Gareth Rushgrove
129. ✓ no droplets exist ✓ provisioner runs successfully ✓ we can count the correct number of hosts ✓ we can list the new hosts ✓ all host serverspec tests pass ✓ clean up all droplets ✓ droplets have been cleaned up ! 7 tests, 0 failures Gareth Rushgrove
130. Gareth Rushgrove garethr/ansible-provisioner
131. Other things to think about (This is just the start)
132. Feature flags for infrastructure Gareth Rushgrove
133. Testing software defined networks Gareth Rushgrove
134. Building a complete infrastructure pipeline Gareth Rushgrove
135. Moving from continuous integration to continuous deployment Gareth Rushgrove
136. Questions? (And thanks for listening)