Puppet Modules: Apps for Ops

Transcript

1. Puppet Modules Apps for Ops! ! Justin Bronn, Esq. @jbronn
2. Slides / Notes https://github.com/jbronn
3. Configuration Management?
4. Complexity
5. <HTML>! <IMG> /cgi-bin/perl.cgi
8. mod_wsgi
9. Don’t Repeat Yourself • Even a “simple” web application has a large number of components • You don’t want to configure them manually
10. Why Puppet?
12. crypt.py
14. Why Puppet? • Mature (in Licensing & Security) • “Explicit is better than Implicit” • Wide OS support: • Linux (of course), Windows, Mac • OpenBSD, Solaris, AIX,HP-UX
15. Installing Puppet
16. Ubuntu 12.04 $ sudo gem install \! --bindir /usr/local/bin! puppet $ sudo apt-get -y install \! libruby libshadow-ruby1.8 rubygems Packages also available: apt.puppetlabs.com
17. OS X $ sudo gem install \! --bindir /usr/local/bin! puppet Package also available: https://downloads.puppetlabs.com/mac/
18. Windows C:\> msiexec /qn /i puppet-3.4.3.msi ^! PUPPET_AGENT_STARTUP_MODE=Disabled! MSI from: https://downloads.puppetlabs.com/windows/
19. Puppet Language
20. Overview • Puppet, the language, is a Ruby DSL • Puppet code lives in files called “manifests”; files end with “.pp” suffix
21. Overview • Catalog: compiled Puppet code, describes relationships between “resources” • Catalog is a an directed acyclic graph • Puppet applies catalog, bringing system into state declared in manifests
22. Resources
23. package {'openssh-server':! ensure => installed,! }

24. package {'openssh-server':! ensure => installed,! }

25. package {'openssh-server':! ensure => installed,! }

26. package {'openssh-server':! ensure => installed,! }

27. package {'openssh-server':! ensure => absent,! }

28. package {'Django':! ensure => '1.6.2',! provider => 'pip',! }

29. Basics
30. Declarative
31. $ssh_dir = '/etc/ssh'! file { $ssh_dir:! ensure => directory,! owner => 'root',! group => 'root',! mode => '0755',! }! $ssh_dir = '/foo/ssh'!
32. $ssh_dir = '/etc/ssh'! file { $ssh_dir:! ensure => directory,! owner => 'root',! group => 'root',! mode => '0644',! }!
33. $array = ['foo', 'bar']! ! $hash = {'key' => 'value'}! $value = $hash['key']! ! $py_versions = split('2.7,3.4', ',')! Data Structures / Function Call
34. $noun = 'Lumberjack'! $sentence = "I'm a ${noun}"! String Interpolation
35. if $python_version == '3.4' {! notice('You have asyncio!')! } ! ! case $python_version {! /^(2.6|2.7)$/: {! }! '3.4': {! }! default: {! fail('Unsupported version!')! }! }! Flow Control
36. Facts
37. $ facter | less! ! $ facter fqdn ipaddress osfamily! fqdn => puppet.local! ipaddress => 10.0.2.15! osfamily => Debian!
38. if $::osfamily == 'RedHat' {! $py_dev_pkg = 'python-devel'! } elsif $::osfamily == 'Debian'{! $py_dev_pkg = 'python-dev'! }! ! ! $:: gives you facts in Puppet
39. if $::osfamily == 'RedHat' {! $py_dev_pkg = 'python-devel'! } elsif $::osfamily == 'Debian'{! $py_dev_pkg = 'python-dev'! }! ! ! $:: gives you facts in Puppet
40. Resource Ordering
41. Resource References Package['openssh-server'] Resource type is capitalized Resource name is in brackets
42. $sshd_config = "${ssh_dir}/sshd_config"! ! file {$sshd_config:! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! content => template('sshd_config.erb'),! require => Package['openssh-server'],! }
 Require Metaparameter
43. $sshd_config = "${ssh_dir}/sshd_config"! ! file {$sshd_config:! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! content => template('sshd_config.erb'),! require => Package['openssh-server'],! }
 Require Metaparameter
44. Package[openssh-server] File[/etc/ssh/sshd_config]
45. service {'ssh':! ensure => running,! enable => true,! subscribe => File[$sshd_config],! }
 Subscribe Metaparameter
46. service {'ssh':! ensure => running,! enable => true,! subscribe => File[$sshd_config],! }
 Subscribe Metaparameter
47. file { $sshd_config:! ...! notify => Service['ssh'],! ...! }
 service {'ssh':! ensure => running,! enable => true,! } Notify Metaparameter
48. file { $sshd_config:! ...! notify => Service['ssh'],! ...! }
 service {'ssh':! ensure => running,! enable => true,! } Notify Metaparameter
49. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows
50. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows
51. Package['openssh-server'] ->! File[$sshd_config] ~>! Service['ssh'] Chaining Arrows
52. Puppet Modules
53. Modules • Containers for Puppet code • Analogous to Python Packages • May be installed from a Puppet “Forge”
54. Installing Modules
55. puppet module install <vendor>-<name>
56. $ puppet module install counsyl-python Notice: Preparing to install into /Users/justin/.puppet/modules ...! Notice: Downloading from https://forge.puppetlabs.com ...! Notice: Installing -- do not interrupt ...! /Users/justin/.puppet/modules! !"# counsyl-python (v0.9.3)! !"# counsyl-sys (v0.9.13)! !"" puppetlabs-stdlib (v4.1.0)! !
57. https://forge.puppetlabs.com https://github.com/jbronn/django-forge
58. Module Structure
59. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/ $ puppet module generate \! counsyl-pyapp
60. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/
61. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/
62. manifests/init.pp class pyapp {! ...! }
 “class” name must match name of module
63. manifests/install.pp class pyapp::install {! ...! }
 Can have other namespaces, but file name must match name of file.
64. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/
65. file { ‘/path/to/settings.py':! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! content => template('pyapp/settings.py.erb'),! } Puppet Templates
66. file { ‘/path/to/settings.py':! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! content => template('pyapp/settings.py.erb'),! } Puppet Templates pyapp/templates/settings.py.erb
67. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/
68. file { '/path/to/foo.js':! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! source => 'puppet:///modules/pyapp/foo.js'! } Puppet Files
69. file { '/path/to/foo.js':! ensure => file,! owner => 'root',! group => 'root',! mode => '0644',! source => 'puppet:///modules/pyapp/foo.js'! } Puppet Files pyapp/files/foo.js
70. counsyl-pyapp/
 Modulefile
 manifests/init.pp
 templates/
 files/
 lib/
71. Writing Modules for Python
72. Roadmap • “Simple” Django application • Needs Celery (and Redis) • Nginx reverse-proxy to Gunicorn
73. class pyapp {! include python! }
 Python / Pip
74. class pyapp {! include python! include python::virtualenv! }
 Virtualenv counsyl-python
75. class pyapp {! include python! include python::virtualenv! include redis! }
 Redis counsyl-redis
76. class pyapp {! include python! include python::virtualenv! include redis! include nginx! }
 Nginx counsyl-nginx
77. Mix & Match
78. Python Types
79. package {'pyapp':! ensure => '0.1.0',! source => 'git+https://github.com …’, provider => 'pip',! }
 pip provider
80. package {'pyapp':! ensure => '0.1.0',! source => 'git+https://github.com …’, provider => 'pip',! }
 pip provider
81. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options => [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
82. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options => [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
83. package {'Django':! ensure => '1.6.2',! provider => 'pipx',! install_options => [! {'--index-url' => ! 'https://pypi.mycorp.com'}! ]! }
 pipx provider
84. venv {'/srv/pyapp': } ! ! ! ! venv type
85. venv {'/srv/pyapp': ! owner => $user,! group => $group,! system_site_packages => true,! }! venv type
86. venv_package {'celery@/srv/pyapp': ! ensure => installed,! }! venv_package type
87. $venv = '/srv/pyapp'! venv_package {"celery@${venv}": ! ensure => installed,! }! venv_package type
88. $venv = '/srv/pyapp'! venv_package {"redis@${venv}": ! ensure => '2.9.1',! require => Class['redis'],! }! venv_package type
89. Class Parameters
90. class pyapp(! $venv = '/srv/pyapp',! $package = 'pyapp',! $source = 'git+https://...',! $version = '0.1.0',! ){! ...! venv { $venv: }! venv_package { "${package}@${venv}":! ensure => $version,! source => $source,! }! ...! }
91. class pyapp(! $venv = '/srv/pyapp'! $package = 'pyapp',! $source = 'git+https://...' ! $version = '0.1.0',! ){! ...! venv { $venv: }! venv_package { "${package}@${venv}":! ensure => $version,! source => $source,! }! ...! }
92. class {'pyapp':! version => '0.8',! } Using Class Parameters
93. class {'pyapp':! version => '0.8',! } Using Class Parameters
94. class {'pyapp':! version => '0.8',! } Using Class Parameters
95. ERB
96. ERB: Ruby Templates • This time, it’s Ruby • Great for when you need dynamic content in a configuration file (e.g., ServerName)
97. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
98. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
99. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
100. DATABASES = {! 'NAME': '<%= @db %>',! }! ! NGINX_CONFIG = '<%= scope['nginx::config_file'] %>'! SERVER_NAME = '<%= scope['::fqdn'] %>'! ! <% ! db_folder = File.dirname(@db_name)! -%>! DB_FOLDER = '<%= db_folder %>'! ERB
101. Running Puppet
102. Stand-Alone
103. puppet apply manifest.pp
104. puppet apply site.pp \! --modulepath /modules Customize Module Path
105. puppet apply -e \! "include pyapp" “Evaluate”
106. puppet agent
107. The Puppetmaster • Centralized server for housing modules and configurations for nodes (servers) • Secured with HTTPS Client Certificates • Nodes must have certificate signed to receive configuration
108. Node Master
109. Node Master node$ sudo puppet agent —test! Info: Creating a new SSL key for node.local! Exiting; no certificate found
110. Node Master master$ sudo puppet cert sign node.local! Info: Signed certificate request for node.local
111. Node Master node$ sudo puppet agent —test! info: Caching certificate for node.local! info: Retrieving plugin! info: Caching catalog for node.local! info: Applying configuration version '1326210629'! notice: Finished catalog run in 0.11 seconds /etc/puppet/modules /etc/puppet/manifests
113. Module Development
114. Use Vagrant!
115. Vagrant • A must-have for DevOps development in general, not just Puppet • Also supports Ansible, Salt, Chef • You may think you can do better…
116. Vagrantfile • Tells Vagrant what to do • An API for provisioning a VM (actually Ruby + magic) • Forward ports, change memory, multiple provisioners, multiple VMs, multiple NICs…
117. Vagrantfile Vagrant.configure('2') do |config|! config.vm.box = "precise64"! config.vm.box_url = "http://..."! ...! end
118. Puppet Provisioner Vagrant.configure('2') do |config|! ...! config.vm.provision "puppet" do |puppet|! puppet.facter = { "vagrant" => "1" }! puppet.module_path = "modules"! end! ...! end
119. Puppet Provisioner Vagrant.configure('2') do |config|! ...! config.vm.provision "puppet" do |puppet|! puppet.facter = { "vagrant" => "1" }! puppet.module_path = "modules"! end! ...! end Looks for manifests/default.pp
120. Another Ruby DSL? • Yeah… • Tremendous workflow benefits • Much easier to share your snowflake environments
121. vagrant up • All end user has to do is type this command • No waiting for a server/sysadmin • Can get to work
122. Base Boxes?
123. Use Packer!
124. Packer • Virtual Machine image creation tool • Written in Go, uses JSON templates • See my sample for creating a base box: • https://github.com/jbronn/packer-vagrant
126. Questions?