Virtualize your stack with Vagrant and Puppet - Columbus Code Camp 2013 Edition

These are the slides from my Virtualizing your stack with Vagrant and Puppet talk, as done at Columbus Code Camp.

For more information on this talk, go to: http://jmather.com/talks/2013-10-12

Transcript

1. Virtualize Your Stack With Vagrant and Puppet Jacob Mather Software Engineer at Mashery
2. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Who am I? What I do: • Software Engineer at Mashery • Co-organizer of the San Francisco PHP User Group • Community Evangelist Where I can be found: • Blog: http://jmather.com • Twitter: @thejmather
3. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Why I am giving this talk
4. Let’s talk about Vagrant
5. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 What is Vagrant? A tool that allows you to create and configure lightweight, reproducible, and portable development environments.
6. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant is built... • in Ruby • to be simple
7. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant terms • Box, or base box, is the ‘initial image’ of a virtual machine. • Host, or your computer, the machine running the VMs • Guest, or vm, referring to an individual VM instance • Provider, an adapter to a VMS (like VirtualBox or VMWare) • Provisioner, provides post-boot configuration of guests
8. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 How to use Vagrant • Install Vagrant - http://vagrantup.com • Check out your repository • Run: vagrant up
9. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrant Commands • vagrant up - Turn guest on • vagrant halt - Turn guest off • vagrant status - Show guest status • vagrant destroy - Delete guest • vagrant suspend - Suspend guest • vagrant resume - Resume guest • vagrant ssh - SSH into guest
10. Let’s talk about code
11. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" end
12. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (add file system) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" config.vm.synced_folder "/data", "/vagrant_data" end
13. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (add port forwarding) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" config.vm.synced_folder "/data", "/vagrant_data" config.vm.network :forwarded_port, guest:80, host: 8080 end
14. Let’s talk about clouds
15. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Replicating production matters If you are writing code that will be deployed to a multiple-host environment, why aren’t you writing your code in a multiple- host environment?
16. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile Vagrant.configure("2") do |config| config.vm.hostname = “box” config.vm.box = "base-box-name" config.vm.box_url = ”http://some/url.box" end
17. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest ready) Vagrant.configure("2") do |config| config.vm.define “box” do |node| node.vm.hostname = “box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end end
18. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest) Vagrant.configure("2") do |config| config.vm.define “box” do |node| node.vm.hostname = “box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end config.vm.define “other-box” do |node| node.vm.hostname = “other-box” node.vm.box = "base-box-name" node.vm.box_url = ”http://some/url.box" end end
19. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest, useful, part 1) nodes = { :'web-server' => { :hostname => 'server.example.com', :ipaddress => '192.168.56.60', }, :'db-server' => { :hostname => 'db.example.com', :ipaddress => '192.168.56.61', } }
20. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (Multi-Guest, useful, part 2) nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm.box = "base-box-name" node.vm.box_url = "http://some/url.box" node.vm.hostname = options[:hostname] node.vm.network :private_network, ip: options[:ipaddress] end end end
21. Let’s talk about Providers
22. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Providers Providers Vagrant
23. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Providers VirtualBox VMWare (WS or Fusion) Parallels Desktop Providers Vagrant
24. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Providers VirtualBox VMWare (WS or Fusion) Parallels Desktop Amazon AWS Digital Ocean Providers Vagrant
25. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Providers (desktop) • VirtualBox is FREE! (sort of) • Some people have trouble with crashes • Some people have trouble with suspend/resume • Built in FS bridging is pretty slow • VMWare (Workstation or Fusion!) is PAID! • You have to buy both VMWare ($49) and provider ($79) • Supposed to be more stable and faster • Parallels Desktop is FREE! • Open source projects FOR THE WIN!
26. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Providers (cloud) • Amazon AWS • Digital Ocean • RackSpace Cloud
27. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Provider Limits Spinning up guests is cool, but what do I do with a box once it’s up? How do I control it? How do I make it useful? Computing hardware without software is nearly useless.
28. Let’s talk about Provisioners
29. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Provisioners Provisioners Vagrant
30. Shell Scripts Puppet Chef Ansible Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Provisioners Provisioners Vagrant
31. Let’s talk about Puppet
32. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 What is Puppet? Puppet is IT automation software that helps system administrators manage infrastructure throughout its lifecycle, from provisioning and configuration to patch management and compliance.
33. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 What?
34. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 What is Puppet in simpler words? Puppet gives you an object oriented way to manage the configuration of your servers, either on your laptop, in the cloud, or in your favorite data center.
35. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet is built... • to be simple • with it’s own declarative language
36. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet terms • Manifests - This is “application” code • Modules - This is “library” code • Templates - Exactly as it sounds • Facter - Environment data • Hiera - Configuration data
37. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 How to use Puppet? Select a base box for Vagrant from Puppet Labs http://puppet-vagrant-boxes.puppetlabs.com/ I tend to use CentOS 6.4 for VirtualBox
38. Let’s talk about code
39. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Vagrantfile (add Puppet) Vagrant.configure("2") do |config| config.vm.box = "base-box-name" config.vm.box_url = "http://some/url.box" config.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests/" puppet.manifest_file = "guest.pp" puppet.module_path = "puppet/modules/" end end
40. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Puppet Manifest (guest.pp) class guest { notify { “Look! My puppet code is working!”: } }
41. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Puppet Manifest (add Apache) class guest { notify { “Look! My puppet code is working!”: } package { “httpd”: ensure => present, } service { “httpd”: require => Package[“httpd”], ensure => running, enabled => true, } }
42. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Basic Puppet Manifest (refactored Apache) # manifests/guest.pp class guest { notify { “Code working!”: } require server::httpd } # modules/server/httpd.pp class server::httpd { package { “httpd”: ensure => present, } service { “httpd”: require => Package[“httpd”], ensure => running, enabled => true, } }
43. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (controlling flow) class guest { notify { “Say me first”: } notify { “Say me second”: require => Notify[“Say me first”], } notify { “Say me third”: require => Notify[“Say me second”], } }
44. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (adding parameters) # modules/server/writetestfile.pp class server::writetestfile ($content) { file { “/tmp/test”: content => $content, } } # modules/server/writetestfiledefault.pp class server::writetestfiledefault ($content = “default”) { file { “/tmp/test”: content => $content } }
45. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (passing parameters) # manifests/guest.pp -- using defaults class guest { include server::writetestfiledefault } # manifests/guest.pp -- passing custom value class guest { class { “server::writetestfiledefault”: content => “test content”, } }
46. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (including classes) class guest { # the class below must be loaded BEFORE guest require server::writetestfiledefault # the class below must be loaded WITH guest include server::writetestfiledefault # the class below must be loaded AFTER guest class { “server::writetestfiledefault”: content => “test content”, } }
47. Let’s talk about how to do things conditionally
48. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 External Input in Puppet Remember there are three primary sources of where data that branches your logic will come from: • User Space Variables • $things = “you set yourself” • Facts • Specific details about your current environment, such as the name of the host, ip address, operating system, etc... • Hiera • Configuration you pass in to enable building for multiple configurations from the same shared codebase
49. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (variables) class guest { # local variable $host = “mybox” if ($host == “mybox”) { notify { “Box!”: } } # Box! if ($host == “mybox”) { notify { “${host}!”: } } # mybox! }
50. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (facts) class guest { # fact - environment configuration if ($::hostname == “box”) { notify { “Box!”: } } # output: Box! if ($hostname == “box”) { notify { “Box!”: } } # output: Box! # assign $hostname locally... $hostname = “some string” if ($::hostname == “box”) { notify { “Box!”: } } # output: Box! if ($hostname == “box”) { notify { “Box!”: } } # no output }
51. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (facts, debugging) class guest { # what facts are there? Let’s find out! # this should be one line, but it is just too long to fit $template = "<%= scope.to_hash.reject { |k,v| !( k.is_a? (String) && v.is_a?(String) ) }.to_yaml %>" # create a file... file { "/tmp/facts.yaml": content => inline_template($template), } }
52. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (if conditionals) class guest { if (“a” == “b”) { notify { “Inconceivable!”: } } elsif (“hello” =~ “help”) { notify { “Regex!”: } } else { notify { “Echo something...”: } } }
53. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (case conditionals) class guest { # $::operatingsystem is a fact, provided directly by Facter case $::operatingsystem { centos, redhat: { $apache = "httpd" } debian, ubuntu: { $apache = "apache2" } default: { fail("Unrecognized operating system") } } case $::operatingsystem { /linux/: { fail(“We don’t support anything with Linux in it”) } } }
54. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Puppet Manifest (selector conditionals) class guest { # $::operatingsystem is a fact, provided directly by Facter $package = $::operatingsystem ? { centos => “httpd”, redhat => “httpd”, /(i?)(ubuntu|debian)/ => “apache2”, default => undef } }
55. Let’s talk about real multiple guest configuration
56. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Configure our guest nodes... nodes = { :'web-server' => { :hostname => 'server.example.com', :ipaddress => '192.168.56.60', }, :'db-server' => { :hostname => 'db.example.com', :ipaddress => '192.168.56.61', } }
57. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Tell Vagrant about them nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm.box = "base-box-name" node.vm.box_url = "http://some/url.box" node.vm.hostname = options[:hostname] node.vm.network :private_network, ip: options[:ipaddress] end end end
58. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Tell Vagrant about Puppet nodes = { … node configuration … } Vagrant.configure("2") do |config| nodes.each_pair do |name,options| config.vm.define name do |node| node.vm... # omit general config node.vm.provision :puppet do |puppet| puppet.manifests_path = "puppet/manifests/" puppet.manifest_file = "guests.pp" puppet.module_path = "puppet/modules/" end end end end
59. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Tell Puppet about the guest nodes class guests { if ($::hostname == “web-server”) { # only run for web server } if ($::hostname == “db-server”) { # only run for db server } }
60. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Tell Puppet about the guest nodes (redux) class guests { if ($::hostname =~ /^web-/) { # only run for web servers } if ($::hostname =~ /^db-/) { # only run for db servers } }
61. Let’s talk about how everything ties together
62. Shell Scripts Puppet Chef Ansible VirtualBox VMWare (WS or Fusion) Parallels Desktop Amazon AWS Digital Ocean Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Vagrant Architecture Providers Provisioners Vagrant
63. Let’s talk about how to convince your boss
64. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Development environments come in all shapes and sizes • Some develop in production – Yes, really. • Some develop on a non production server – Not a copy of production, not by a long shot • Some develop locally on a “non production server” – Home grown solutions that “work” until they don’t • Some develop on a development server – Actually tries to be like production – Not always like production – May have additional tuning to help with developmenty things • Some develop in a true development environment – Locally or remote, coming soon to a computer near you
65. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Why developing in production is really bad Assuming, of course, that you are testing your code before you make it live. Production should always move from one stable version of code to another stable version of code. When you edit code in production, you inherently introduce instability, and can no longer be assured that your code is running as tested.
66. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Why developing on a non production setup is also bad Debugging environmental issues is slow and painful. Developers should be solving problems with software, not debugging configuration issues. When you build your code in an environment that isn’t related to production in any tangible way, you will often discover just how different your development environment is from production.
67. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Where your development environment lives Remote • Easy for operations to set up • Easy to enforce a rigid structure • Limits developer’s choice of tool chain selection • Can get expensive and complicated to maintain • Developers must be online to do any development Local • Can require beefier hardware • Can take extra work to set up • Developers have the freedom to work how they work best • Developers can assist with infrastructural initiatives
68. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Bad development environments will sap the resources out of your team. In addition to the time spent developing and testing new code and fixing bugs, you will spend time: When development environments attack Debugging broken environments or settings Debugging broken behavior between dev and prod Debugging broken behavior between two dev systems Working in broken ways because “it works”
69. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Failure to specifically build an environment for development will directly impact developer productivity and product quality.
70. Let’s talk about what you should remember
71. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Takeaways • Bad (or unstable) environments (be it dev, qa, staging, or prod) hurt developers. • Puppet lets you automate all of the configuration of your systems (dev, qa, staging, and prod), ensuring consistency between them. • Vagrant is easy to use, and will let you use your production puppet configs to build development environments. • You will manage server configuration, whether you decide to or not. If you choose not to do it formally it will come back to bite you when you forget something later.
72. Let’s talk about where to go from here
73. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 If you’re still not sold... h"p://PuPHPet.com/ / PuPHPet/provides/a/ simple/config/tool/for/ those/who/don’t/want/ to/=nker/with/puppet/ and/need/a/LAMP/stack/ up/and/running/NOW./ V2 was released this morning!!!
74. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 And also look at... • Packer • http://packer.io • Take your Vagrant and Puppet usage to the next level by using Packer to pre-provision images for easy first spins • Build prebuilt systems for multiple VMS and cloud platforms • Docker • http://docker.io • A different idea of how to build multiple machine VM environments
75. Virtualize your environment with Vagrant and Puppet - Columbus Code Camp - October 12th, 2013 Thank you! More information about this talk can be found at: http://jmather.com/talks/2013-10-12