Intro to Chef - Rocky Mountain Ruby

Transcript

1. An Introduction to Chef Nathen Harvey @nathenharvey Joshua Timberman @jtimberman nharvey@opscode.com github.com/nathenharvey joshua@opscode.com github.com/jtimberman
2. Introductions
3. Who Are We?
4. Who Are You? • System administrator? • Software developer/engineer? • Used Chef before?
5. Tutorial Objectives • Describe the types of problems Chef solves • List the components of Chef • Create a new cookbook • Use some primitives of Chef recipes • Follow some common workflows used by experienced Chef users/developers
6. Expectations • This is a half-day workshop, not a comprehensive course. • We will do some hands on exercises. • You should get a taste for automating with Chef. • You should have received instructions for prerequisites prior to this tutorial • Chef works and runs on Windows, but the exercises for this tutorial are Linux-based
7. We have a problem...
8. http://www.flickr.com/photos/michaelheiss/3090102907/ Complexity
9. Items of Manipulation (Resources) • Nodes • Networking • Files • Directories • Symlinks • Mounts • Routes • Users • Groups • Packages • Services • Filesystems
10. Application A tale of growth...
11. Application Application Database Add a database
12. Application App Databases Make database redundant
13. App Servers App Databases Application server redundancy
14. App LB App Servers App Databases Add a load balancer
15. App LBs App Servers App Databases Webscale!
16. App LBs App Servers App DB Cache App DBs Now we need a caching layer
17. App LBs App Servers App DB Cache App DBs Infrastructure Has a Topology
18. Round Robin DNS App Servers App DB Cache App DBs Floating IP? Your's Is a Snowflake
19. App LBs App Servers < Shiny! DB slaves Cache DB Cache DBs Complexity Increases Quickly Are we monitoring??
21. The Chef Framework • Reasonability • Flexibility • Library & Primitives • TIMTOWTDI • Sane defaults http://www.flickr.com/photos/wonderlane/3609342683/sizes/l/in/photostream/
22. The Chef Tool(s) • ohai • chef-client • chef-shell • knife • The Ruby language Omnibus - Full Stack Native Packages
23. The Chef API • HTTPS, RESTful API w/ JSON, RSA key auth • Infrastructure data store such as node data • Search Service • Derivative Services? http://www.flickr.com/photos/core-materials/4419853626/sizes/o/in/photostream/
24. The Chef Community • Chef is Open Source! • Apache License, Version 2.0 • Hundreds of Individual and Corporate contributors. • Hundreds of cookbooks available from the community • http://community.opscode.com
25. How does it work? http://i3.kym-cdn.com/photos/images/original/000/046/123/magnets.jpg
26. Chef is Infrastructure as Code • Programmatically provision and configure • Treat like any other code base • Reconstruct business from code repository, data backup, and bare metal resources. http://www.flickr.com/photos/louisb/4555295187/
27. Resources and Recipes
28. package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end Declarative abstraction to system resources
29. package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end
30. Resources describe what Not how.
31. Resources take action through Providers
32. Providers perform the how
33. def install_package(name, version) package_name = "#{name}=#{version}" package_name = name if @is_virtual_package run_command_with_systems_locale( :command => "apt-get -q -y #{expand_options(@new_resource.options)} install #{package_name}", :environment => { "DEBIAN_FRONTEND" => "noninteractive" } ) end Provider Example
34. Package Resource package "git" { yum install git apt-get install git pacman sync git pkg_add -r git Providers are determined by node's platform
35. Recipes are collections of resources package "apache2" do action :install end template "/etc/apache2/apache2.conf" do source "apache2.conf.erb" owner "www-data" group "www-data" mode 00644 notifies :restart, "service[apache2]" end service "apache2" do supports :status => true, :restart => true action [:enable, :start] end
36. Cookbooks • Cookbooks are collections of Recipes • Cookbooks contain related components • Files, Templates, Libraries • A cookbook is responsible for configuring a single thing, e.g. • apache2 • postgresql • A recipe is responsible for a component, e.g. • api • server • client
37. Roles • Roles describe nodes • Roles indicate functionality • Roles have a first class API primitive
38. http://www.flickr.com/photos/peterrosbjerg/3913766224/ Chef Nodes • Chef runs on nodes • Chef nodes do the heavy lifting • Authority about themselves • Stored on the server when using Chef Server • Indexed for search
39. Search • Search for nodes with Roles • Find Topology Data • IP addresses • Hostnames • FQDNs http://www.flickr.com/photos/kathycsus/2686772625
40. Hands On Exercises
41. Hands on Exercises • The majority of the hands on exercises will be related to creating an "apache" cookbook. • The goals are to learn elements of Chef, not to learn Apache. We're going to do things the hard way, by typing in a lot of code. We have some gists with large sections of code already available to reduce what you need to type in certain places. • Errors and typos are good, as they will help students learn how to resolve errors.
42. Legend
43. Legend: Do I run that command on my workstation? $ whoami i-am-a-workstation This is an example of a command to run on your workstation user@hostname:~$ whoami i-am-a-chef-node This is an example of a command to run on your target node via SSH.
44. $ ifconfig Legend: Example Terminal Command and Output lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> mtu 16384 ! options=3<RXCSUM,TXCSUM> ! inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1 ! inet 127.0.0.1 netmask 0xff000000 ! inet6 ::1 prefixlen 128 gif0: flags=8010<POINTOPOINT,MULTICAST> mtu 1280 stf0: flags=0<> mtu 1280 en0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500 ! ether 28:cf:e9:1f:79:a3 ! inet6 fe80::2acf:e9ff:fe1f:79a3%en0 prefixlen 64 scopeid 0x4 ! inet 10.100.0.84 netmask 0xffffff00 broadcast 10.100.0.255 ! media: autoselect ! status: active p2p0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> mtu 2304 ! ether 0a:cf:e9:1f:79:a3 ! media: autoselect ! status: inactive
45. OPEN IN EDITOR: SAVE FILE! ~/hello_world Hi! I am a friendly file. Legend: Example of editing a file on your workstation
46. Workstation Setup Getting started
47. Objectives • Have Chef installed on your Workstation • Have an empty chef-repo • Have a unique User created • Have an Organization created for use during training • Understand what Knife is • Have a working Knife configuration
48. Landscape of Chef-managed Infrastructure
49. Workstation Setup • Install Chef (if not already installed) • https://www.opscode.com/ chef/install/
50. $ curl -L http://www.opscode.com/chef/install.sh | sudo bash Workstation Setup - Mac OS X / Linux % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed 100 6515 100 6515 0 0 20600 0 --:--:-- --:--:-- --:--:-- 31172 Downloading Chef for ubuntu... Installing Chef Selecting previously unselected package chef. (Reading database ... 47446 files and directories currently installed.) Unpacking chef (from .../tmp.MqRJP6lz/chef__amd64.deb) ... Setting up chef (11.4.4-2.ubuntu.11.04) ... Thank you for installing Chef! Processing triggers for initramfs-tools ... update-initramfs: Generating /boot/initrd.img-3.2.0-48-virtual
51. Workstation Setup - Windows • Windows • 2008 (Windows 7) or 2012 (Windows 8) • i686 (32-bit) or x86_64 (64-bit) • 11.6.0 Download and install this file
52. Your Chef Server for this class... • Set up Enterprise Chef account • https://getchef.opscode.com/signup
53. Create new account • Sign up for a new account
54. Create new account • Fill in the form with your own information
55. Download “Starter Kit” • You get a .zip file from clicking this • Unzip the zipfile - you’ll get a “chef- repo” • Put the “chef-repo” somewhere, e.g.: • C:\Users\you\chef-repo (Win) • /Users/you/chef-repo (Mac) • /home/you/chef-repo (Linux)
56. A quick tour of the chef-repo • Every infrastructure managed with Chef has a Chef Repository (“chef-repo”) • Let’s see what’s inside the chef-repo...
57. $ cd chef-repo Verify that knife is working [~/chef-repo]$
58. $ ls -al A quick tour of the chef-repo total 40 drwxr-xr-x 11 nathenharvey staff 374 Sep 23 09:44 . drwxr-xr-x+ 86 nathenharvey staff 2924 Sep 23 09:45 .. drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 .berkshelf drwxr-xr-x 5 nathenharvey staff 170 Sep 23 2013 .chef -rw-r--r--@ 1 nathenharvey staff 495 Sep 23 2013 .gitignore -rw-r--r--@ 1 nathenharvey staff 1433 Sep 23 2013 Berksfile -rw-r--r--@ 1 nathenharvey staff 2292 Sep 23 2013 README.md -rw-r--r--@ 1 nathenharvey staff 3562 Sep 23 2013 Vagrantfile -rw-r--r--@ 1 nathenharvey staff 588 Sep 23 2013 chefignore drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 cookbooks drwxr-xr-x 3 nathenharvey staff 102 Sep 23 2013 roles
59. $ ls .chef What’s inside the .chef directory? ORGNAME-validator.pem USERNAME.pem knife.rb
60. What’s inside the .chef directory? • knife.rb is the configuration file for Knife. • The other two files are certificates for authentication with the Chef Server • We’ll talk more about that later.
61. • Your version may be different, that’s ok! Verify Knife $ knife --version Chef: 11.6.0 $ knife client list ORGNAME-validator
62. Knife is the command-line tool for Chef • Knife provides an interface between a local Chef repository and the Chef Server. Knife lets you manage: • Nodes • Cookbooks and recipes • Roles • Stores of JSON data (data bags), including encrypted data • Environments • Cloud resources, including provisioning • The installation of Chef on management workstations • Searching of indexed data on the Chef Server
63. Test Node Setup Setting up a target
64. Objectives • Have Chef installed on your test node via “knife bootstrap” • Understand how knife bootstrap configures a node to use the Organization created in the previous section • Understand the basic configuration needed to run chef-client
65. Target Instances • ec2-based Instance • Ubuntu 12.04.2 LTS • SSH • Username: opscode • Password: opscode
66. $ knife bootstrap IPADDRESS --sudo -x opscode -P opscode -N “target1” "Bootstrap" the Target Instance Bootstrapping Chef on target1 target1 Starting Chef Client, version 11.6.0
67. local workstation managed node (VM) chef-client $ knife bootstrap IPADDRESS --sudo -x USERNAME -P PASSWORD Opscode Enterprise Chef chef_server_url validation_client_name validation_key SSH! bash -c ' install chef configure client run chef'
68. Verify Your Target Instance’s Chef-Client is Configured Properly $ ssh opscode@IPADDRESS opscode@target1:~$ ls /etc/chef client.pem client.rb first-boot.json validation.pem opscode@target1:~$ which chef-client /usr/bin/chef-client
69. opscode@target1:~$ cat /etc/chef/client.rb Examine /etc/chef/client.rb log_level :auto log_location STDOUT chef_server_url "https://api.opscode.com/organizations/ ORGNAME" validation_client_name "ORGNAME-validator" node_name "target1"
70. Overview of our environment
72. Chef Server • We're using hosted Enterprise Chef
73. Managed Nodes • Many nodes can be managed by a Chef Server • We're going to manage a single node
74. Workstation • We write Chef code on a local workstation like a laptop • We upload that code to the Chef Server • Using the API
75. Create an apache cookbook
76. Objectives • Understand what a cookbook is • Know how to create a new cookbook • Understand what a recipe is • Understand how to use the package, service, and template resources • Know how to upload a cookbook to the Chef Server • Understand what a run list is, and how to set it • How to read the output of the chef-client run
77. What is a cookbook? • A cookbook is like a "package" for Chef recipes • It contains all the recipes, files, templates, libraries, etc. required to configure a portion of your infrastructure • Typically cookbooks map 1:1 to a piece of software or functionality. • "our tomcat cookbook" • "our zlib cookbook" • "our security_policy cookbook"
78. $ knife cookbook create apache Create a new Cookbook ** Creating cookbook apache ** Creating README for cookbook: apache ** Creating CHANGELOG for cookbook: apache ** Creating metadata for cookbook: apache
79. $ ls -la cookbooks/apache Explore the cookbook total 24 drwxr-xr-x 13 opscode opscode 442 Mar 24 21:25 . drwxr-xr-x 5 opscode opscode 170 Mar 24 21:25 .. -rw-r--r-- 1 opscode opscode 412 Mar 24 21:25 CHANGELOG.md -rw-r--r-- 1 opscode opscode 1447 Mar 24 21:25 README.md drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 attributes drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 definitions drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 files drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 libraries -rw-r--r-- 1 opscode opscode 276 Mar 24 21:25 metadata.rb drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 providers drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 recipes drwxr-xr-x 2 opscode opscode 68 Mar 24 21:25 resources drwxr-xr-x 3 opscode opscode 102 Mar 24 21:25 templates
80. $ cat cookbooks/apache/metadata.rb Read the metadata name 'apache' maintainer 'YOUR_COMPANY_NAME' maintainer_email 'YOUR_EMAIL' license 'All rights reserved' description 'Installs/Configures apache' long_description IO.read(File.join(File.dirname(__FILE__), 'README.md')) version '0.1.0'
81. Cookbook Metadata • Cookbooks are like packages of configuration for the infrastructure • They're artifacts that have a name and a version • Metadata can also have information about the cookbook like its license and maintainer • Cookbooks can depend on other cookbooks, too
82. Other relevant components • We'll use other components of the cookbook throughout the exercises •recipes/default.rb •templates/default/ •attributes/[default.rb]
83. What is a Recipe? • Ordered list of Resources • Ruby Domain-Specific Language (DSL)
84. OPEN IN EDITOR: cookbooks/apache/recipes/default.rb # # Cookbook Name:: apache # Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # Edit the default recipe
85. SAVE FILE! # # Cookbook Name:: apache # Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "apache2" do action :install end Exercise: Add a package resource to install Apache to the default recipe OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
86. package "apache2" do action :install end So the resource we just wrote...
87. package "apache2" do action :install end So the resource we just wrote... • Is a package resource
88. package "apache2" do action :install end So the resource we just wrote... • Is a package resource • Whose name is apache2
89. package "apache2" do action :install end So the resource we just wrote... • Is a package resource • Whose name is apache2 • With an install action
90. How does the package install? • Resources are declarative - we say what we want to have happen, rather than how • Chef uses what platform the node is running to determine the correct provider for a resource
91. SAVE FILE! ... # All rights reserved - Do Not Redistribute # package "apache2" do action :install end service "apache2" do action [ :enable, :start ] end Exercise: Add a service resource to ensure the service is started and enabled at boot OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
92. service "apache2" do action [ :enable, :start ] end So the resource we just wrote...
93. service "apache2" do action [ :enable, :start ] end So the resource we just wrote... • Is a service resource
94. service "apache2" do action [ :enable, :start ] end So the resource we just wrote... • Is a service resource • Whose name is apache2
95. service "apache2" do action [ :enable, :start ] end So the resource we just wrote... • Is a service resource • Whose name is apache2 • With two actions: start and enable
96. Order Matters • The order you write resources in a recipe is the order they will be executed in
97. SAVE FILE! ... service "apache2" do action [ :enable, :start ] end template "/var/www/index.html" do source "index.html.erb" mode "0644" end Add a template resource to write the home page OPEN IN EDITOR: cookbooks/apache/recipes/default.rb
98. So the resource we just wrote... template "/var/www/index.html" do source "index.html.erb" mode "0644" end
99. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the resource we just wrote... • Is a template resource
100. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the resource we just wrote... • Is a template resource • Whose name is /var/www/index.html
101. template "/var/www/index.html" do source "index.html.erb" mode "0644" end So the resource we just wrote... • Is a template resource • Whose name is /var/www/index.html • With two parameters: • source of index.html.erb • mode of “0644”
102. Best Practice: Omit the action if it is the default • Has no action! • If you omit the action in Chef, we default to the most common positive action. In this case, it is the :create action. template "/var/www/index.html" do source "index.html.erb" mode "0644" end
103. Full contents of the apache recipe # # Cookbook Name:: apache # Recipe:: default # # Copyright 2013, YOUR_COMPANY_NAME # # All rights reserved - Do Not Redistribute # package "apache2" do action :install end service "apache2" do action [ :enable, :start ] end template "/var/www/index.html" do source "index.html.erb" mode "0644" end
104. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, world!</p> Add an index template
105. $ knife cookbook upload apache Exercise: Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
106. What is "knife cookbook upload"? • Several knife commands correspond to API endpoints • The verb indicates what we're doing with the cookbook • In this case we upload the cookbook to the server through the API. It's stored by the server... • We can show information about it too...
107. Checkpoint • We have a cookbook named apache • Our apache cookbook has a default recipe • It has a template source file, index.html.erb • We have uploaded the cookbook to the Chef server.
108. Anatomy of a Chef Run
109. opscode@target1:~$ sudo chef-client Run the chef-client on your test node [sudo] password for opscode: Starting Chef Client, version 11.6.0 resolving cookbooks for run list: [] Synchronizing Cookbooks: Compiling Cookbooks... [2013-09-23T14:31:38+00:00] WARN: Node target1 has an empty run list. Converging 0 resources Chef Client finished, 0 resources updated
110. Wait, why didn't Chef apply the recipe? • The node has an empty run list • Let us look at what we have now...
111. $ knife node show target1 View details about the node Node Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: Roles: Recipes: Platform: ubuntu 12.04 Tags:
112. $ knife node run list add target1 “recipe[apache]” Add apache to the run list target1: run_list: recipe[apache]
113. opscode@target1:~$ sudo chef-client Run the chef-client on your test node Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install - install version 2.2.22-1ubuntu1 of package apache2 * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from 94850c to 463a6b --- /var/www/index.html 2013-09-23 14:34:24.378187238 +0000 +++ /tmp/chef-rendered-template20130923-3442-1gji5jy 2013-09-23 14:34:28.174187236 +0000 @@ -1,4 +1 @@ -<html><body><h1>It works!</h1> -<p>This is the default web page for this server.</p> -<p>The web server software is running but no content has been added, yet.</p> -</body></html> +<p>Hello, world</p> Chef Client finished, 2 resources updated
114. * service[apache2] action enable (up to date) * service[apache2] action start (up to date) Why wasn't the service updated? • Chef takes idempotent actions on resources to ensure they are converged to the desired state • The apache2 package on Ubuntu automatically enables and starts the apache2 service in its post- install script • Thus, Chef didn't need to do anything
115. If we stopped apache2... $ sudo /etc/init.d/apache2 stop * Stopping web server apache2 apache2: [ OK ] $ sudo chef-client .... * service[apache2] action start - start service service[apache2]
116. build node authenticate sync cookbooks load cookbooks converge node.save notification handlers exception Yes No chef-client success? expanded run list (recipes) Ohai! node_name platform platform_version
117. Processing Recipes is Two-Phase • Chef processes recipes in two phases during "convergence" • Evaluate all the Ruby code in the recipe, looking for "Chef Resources" • Execute the providers' actions to put each resource in the declared state
118. Node Attributes
119. The Chef Node Object • Nodes are the objects that you manage with Chef • They have a few different properties • attributes • run_list • chef_environment
120. The Chef Node Object • In client/server Chef, the Chef Server stores node object data • It becomes searchable through the API with knife and with recipes • Some of the data comes from ohai, which takes an inventory of the system and emits JSON data • You can add data to the node through attributes in cookbooks, roles, directly on a node, etc
121. Node Attributes Syntax • Node attributes are hashes (of hashes) • Reference hash keys with square brackets and keys as strings •node["hostname"] •node["kernel"]["machine"]
122. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, world!</p> <p>My name is <%= node['hostname'] %></p> Update the home page template
123. $ knife cookbook upload apache Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
124. opscode@target1:~$ sudo chef-client Run the chef-client on your test node Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from 463a6b to b50d3b --- /var/www/index.html 2013-09-23 14:34:28.174187236 +0000 +++ /tmp/chef-rendered-template20130923-4337-1csr3dp 2013-09-23 14:39:56.426187239 +0000 @@ -1 +1,2 @@ -<p>Hello, world</p> +<p>Hello, world!</p> +<p>My name is ip-10-164-9-210</p> Chef Client finished, 1 resources updated
125. opscode@target1:~$ ohai Run Ohai on the node { "languages": { "ruby": { }, "perl": { "version": "5.14.2", "archname": "x86_64-linux-gnu-thread-multi" }, "python": { "version": "2.7.3", "builddate": "Aug 1 2012, 05:14:39" } }, <SNIP>
126. opscode@target1:~$ ohai hostname Run Ohai on the node [ "ip-10-164-9-210" ]
127. $ knife node show target1 Show the node object Node Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: recipe[apache] Roles: Recipes: apache Platform: ubuntu 12.04 Tags:
128. $ knife node show target1 -a hostname Show specific node attribute target1: hostname: ip-10-164-9-210
129. Cookbook Attributes
130. Objectives • Set node attributes from a cookbook • Understand node attribute precedence
131. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/attributes/default.rb default["apache"]["greeting"] = "World" Create an attributes file
132. Cookbook Attributes & Precedence • Always set default node attributes in your cookbooks' attributes files • Use "sane" defaults - no surprises • You can use attributes in roles to set new values • Roles take precedence over cookbook settings • When a value must be set to a certain value, use override, but use this sparingly • You can’t override ohai's (automatic) attributes!
133. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p> <p>My name is <%= node['hostname'] %></p> Update the home page template
134. $ knife cookbook upload apache Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
135. opscode@target1:~$ sudo chef-client Run the chef-client on your test node Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * template[/var/www/index.html] action create - update content in file /var/www/index.html from b50d3b to 764f54 --- /var/www/index.html 2013-09-23 14:39:56.426187239 +0000 +++ /tmp/chef-rendered-template20130923-4826-1me21yb 2013-09-23 14:54:37.974187238 +0000 @@ -1,2 +1,2 @@ -<p>Hello, world!</p> +<p>Hello, World!</p> <p>My name is ip-10-164-9-210</p> Chef Client finished, 1 resources updated
136. Checkpoint • We have a node attribute • node["apache"]["greeting"] • We've updated the index.html template to use this attribute. • This will be used again soon!
137. Data Bags
138. Objectives • Use Data Bags for data-driven recipes • Use multiple recipes for a node's run list • Control execution of arbitrary commands with Chef's resource conditionals
139. Data Bags are generic stores of information • Data bags are generic, arbitrary stores of information about the infrastructure. • Data Bag Items are JSON data • Our apache cookbook provides a good baseline • We'll drive site-specific virtual hosts with data bags
140. $ mkdir -p data_bags/vhosts Create a directory for Data Bags
141. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/bears.json { "id" : "bears", "port" : 80 } Add a Data Bag Item
142. OPEN IN EDITOR: SAVE FILE! data_bags/vhosts/clowns.json { "id" : "clowns", "port" : 81 } Add a Data Bag Item
143. $ knife upload data_bags/vhosts Upload the data bags Created data_bags/vhosts Created data_bags/vhosts/bears.json Created data_bags/vhosts/clowns.json
144. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/default.rb service "apache2" do action [:start, :enable] end execute "a2dissite default" do only_if do File.symlink?("/etc/apache2/sites-enabled/000-default") end notifies :restart, "service[apache2]" end template "/var/www/index.html" do Disable the default apache site
145. template "/var/www/index.html" do source "index.html.erb" mode "0644" end OPEN IN EDITOR: /cookbooks/apache/recipes/default.rb SAVE FILE! Delete the template resource • Remove this resource from the recipe
146. $ knife diff cookbooks/apache Diff the cookbook --- cookbooks/apache/recipes/default.rb 2013-09-23 11:46:22.000000000 -0400 +++ cookbooks/apache/recipes/default.rb 2013-09-23 11:46:22.000000000 -0400 @@ -14,7 +14,9 @@ action [:start, :enable] end -template "/var/www/index.html" do - source "index.html.erb" - mode "0644" -end +execute "a2dissite default" do + only_if do + File.symlink?("/etc/apache2/sites-enabled/000-default") + end + notifies :restart, "service[apache2]" +end
147. $ knife cookbook upload apache Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
148. $ knife diff cookbooks/apache Diff the cookbook
149. A new recipe for virtual hosts • We'll create an apache::vhosts recipe to manage the virtual hosts we created in data bag items • There's a number of new things to talk about in this recipe • We'll take this nice and slow :)
150. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb data_bag("vhosts").each do |site| site_data = data_bag_item("vhosts", site) site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" end Create a vhosts recipe
151. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb document_root = "/srv/apache/#{site_name}" template "/etc/apache2/sites-available/#{site_name}" do source "custom-vhost.erb" mode "0644" variables( :document_root => document_root, :port => site_data["port"] ) end end Add a Virtual Hosts Configuration Template
152. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end execute "a2ensite #{site_name}" do not_if do ::File.symlink?("/etc/apache2/sites-enabled/#{site_name}") end notifies :restart, "service[apache2]" end end Enable the Virtual Hosts
153. not_if and only_if • The not_if parameter causes the resource’s actions to be taken only if its argument returns false • The only_if parameter is the opposite of not_if - the actions are taken only if the arguments return true • Both not_if and only_if are part of Chef (resources), not part of Ruby
154. not_if and only_if • not_if and only_if parameters take either a string, or a Ruby block argument (do..end or {..}) • When the argument is a string, Chef evaluates it as a shell command to run. • When the argument is a Ruby block, Chef evaluates it as Ruby code to execute. • This is the equivalent to the code we wrote: not_if "test -L /etc/apache2/sites-enabled/#{site_name}"
155. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end directory document_root do mode "0755" recursive true end end Add a directory resource
156. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb end template "#{document_root}/index.html" do source "index.html.erb" mode "0644" variables( :site_name => site_name, :port => site_data["port"] ) end end Index for each vhost https://gist.github.com/6673845
157. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/custom-vhosts.erb <% if @port != 80 -%> Listen <%= @port %> <% end -%> <VirtualHost *:<%= @port %>> ServerAdmin webmaster@localhost DocumentRoot <%= @document_root %> <Directory /> Options FollowSymLinks AllowOverride None </Directory> <Directory <%= @document_root %>> Options Indexes FollowSymLinks MultiViews AllowOverride None Order allow,deny allow from all </Directory> </VirtualHost> Index for each vhost https://gist.github.com/2866454
158. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/templates/default/index.html.erb <p>Hello, <%= node['apache']['greeting'] %>!</p> <p>My name is <%= node['hostname'] %></p> <p>We love <%= @site_name %></p> <p>Served from <%= node['ipaddress'] %>:<%= @port %></p> Update the index.html template
159. $ knife diff cookbooks/apache Diff the cookbook diff --knife cookbooks/apache/recipes/default.rb cookbooks/apache/recipes/default.rb --- cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400 +++ cookbooks/apache/recipes/default.rb 2013-09-23 12:04:18.000000000 -0400 @@ -19,4 +19,4 @@ File.symlink?("/etc/apache2/sites-enabled/000-default") end notifies :restart, "service[apache2]" -end \ No newline at end of file +end diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/recipes/vhosts.rb new file --- /dev/null 2013-09-23 12:04:18.000000000 -0400 +++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:04:18.000000000 -0400 @@ -0,0 +1,35 @@ +data_bag("vhosts").each do |site|
160. $ knife cookbook upload apache Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
161. opscode@target1:~$ sudo chef-client Run the chef-client on your test node Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 3 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run - execute a2dissite default * service[apache2] action restart - restart service service[apache2] Chef Client finished, 2 resources updated
162. Oops!
163. $ knife node run list add target1 "recipe[apache::vhosts]" Update the run list target1: run_list: recipe[apache] recipe[apache::vhosts]
164. opscode@target1:~$ sudo chef-client Run the chef-client on your test node Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 11 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears] action create - create new file /etc/apache2/sites-available/bears - update content in file /etc/apache2/sites-available/bears from none to bf6a12 --- /etc/apache2/sites-available/bears 2013-09-23 16:11:07.130187239 +0000 +++ /tmp/chef-rendered-template20130923-5699-1w4prah 2013-09-23 16:11:07.130187239 +0000 @@ -0,0 +1 @@
165. Think about what we just did... • We had two virtual hosts... • But we could arbitrarily add more... • Tigers on port 82, Lions on port 83, oh my!
166. Checkpoint • Our cookbook has two recipes, default and vhosts • Additional data bags can be added, expanding our Virtual Hosting empire!
167. Roles
168. What is a role? • So far, we’ve been just adding recipes directly to our single node's run list • But that’s not how infrastructure works - think about how you refer to servers • "It’s a web server" • "It’s a database server" • or, "It's a database-master server"... • "It’s a monitoring server"
169. How you use roles • Roles allow you to conveniently encapsulate the run lists and attributes required for a server to "be" what you already think it is • In practice, roles make it easy to configure many nodes identically without repeating yourself each time • Roles are a first class API primitive on the Chef Server
170. Best Practice • Like data bags, you have options for creating roles • The best practice is that all of your roles live in the roles directory of your chef-repo • They can be created via the API and knife, but having them in source control gives you the history of changes
171. OPEN IN EDITOR: SAVE FILE! roles/webserver.json { "name" : "webserver", "default_attributes" : { "apache" : { "greeting" : "Rocky Mountain Ruby" } }, "run_list" : [ "recipe[apache]", "recipe[apache::vhosts]" ] } Create a webserver role
172. Components of a role • Roles must have a name • Roles may have a description • Roles may have a run_list, just like a node • Roles may set node attributes • default_attributes • override_attributes http://docs.opscode.com/essentials_roles.html
173. $ knife upload roles/webserver.json Upload the role Created roles/webserver.json
174. $ knife node run list remove target1 "recipe[apache::vhosts]" Remove the vhosts recipe target1: run_list: recipe[apache]
175. $ knife node run list remove target1 "recipe[apache]" Remove the apache recipe target1: run_list:
176. $ knife node run list add target1 "role[webserver]" Add the webserver role target1: run_list: role[webserver]
177. opscode@target1:~$ sudo chef-client Run chef-client * template[/srv/apache/bears/index.html] action create - update content in file /srv/apache/bears/index.html from 509865 to bee814 --- /srv/apache/bears/index.html 2013-09-23 16:11:07.398187242 +0000 +++ /tmp/chef-rendered-template20130923-6418-om7gju 2013-09-23 16:29:27.778187238 +0000 @@ -1,4 +1,4 @@ -<p>Hello, World!</p> +<p>Hello, Rocky Mountain Ruby!</p> <p>My name is ip-10-164-9-210</p> <p>We love bears</p> <p>Served from 10.164.9.210:80</p> * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create - update content in file /srv/apache/clowns/index.html from 2055c3 to 2290a0 --- /srv/apache/clowns/index.html 2013-09-23 16:11:07.654187238 +0000 +++ /tmp/chef-rendered-template20130923-6418-j7lsw6 2013-09-23 16:29:27.862187239 +0000 @@ -1,4 +1,4 @@
178. Attributes can be set multiple places • This is for flexibility • Set a "sane default" that will be used in a cookbook • Override it easily in a role (higher priority) • In all, there are 15 places where attributes come from (!!) • In practice, you'll use 2-3 most of the time. • The others are there when you need them. • http://docs.opscode.com/chef_overview_attributes.html
179. Data Bags Are Not Attributes • Important: data bag items are not attributes • Data bags are a separate API end point • Data bags are not tied to a specific node or role in the infrastructure • Not even necessarily tied to anything, just data you want to store
180. Checkpoint • We now have a webserver role. • We could apply just this role on more nodes to scale out our Virtual Hosting service. • Roles are a great way to assign attributes for specific purposes
181. Search
182. Chef's Search Feature • Search ties together the infrastructure topology • We can now search the Chef Server for all the "webserver" nodes • This is relevant for a variety of reasons • Load balancing several front ends (search from nginx, or haproxy for example) • Monitoring HTTP (search from nagios cookbook) • Graphing traffic (search from munin cookbook)
183. However, we only have one node... • We don't have an environment that is really setup for doing search • But we will talk about what kind of search we do with the command-line, or in a recipe
184. $ knife search node "role:webserver" Search for webservers with knife 1 items found Node Name: target1 Environment: _default FQDN: ip-10-164-9-210.ec2.internal IP: 54.226.122.99 Run List: role[webserver] Roles: webserver Recipes: apache, apache::vhosts Platform: ubuntu 12.04 Tags:
185. Search from a recipe webservers = search(:node, "role:webserver") # or search(:node, "role:webserver").each do |web| #things do do on all web servers... end
186. Available search indexes • The Chef Server indexes JSON data for the major API endpoints: • node • client • environment • role • All data bags are also indexed for search... • vhosts (our example)
187. $ knife search vhosts "port:81" Search for webservers with knife 1 items found chef_type: data_bag_item data_bag: vhosts id: clowns port: 81
188. Replace Data Bag Item Lookup with Search
189. OPEN IN EDITOR: SAVE FILE! cookbooks/apache/recipes/vhosts.rb search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}" Refactor the vhosts recipe
190. $ knife diff cookbooks/apache View the diff diff --knife cookbooks/apache/recipes/vhosts.rb cookbooks/apache/ recipes/vhosts.rb --- cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000 -0400 +++ cookbooks/apache/recipes/vhosts.rb 2013-09-23 12:38:22.000000000 -0400 @@ -1,5 +1,4 @@ -data_bag("vhosts").each do |site| - site_data = data_bag_item("vhosts", site) +search(:vhosts).each do |site_data| site_name = site_data["id"] document_root = "/srv/apache/#{site_name}"
191. $ knife cookbook upload apache Upload the cookbook Uploading apache [0.1.0] Uploaded 1 cookbook.
192. opscode@target1:~$ sudo chef-client Run chef-client Starting Chef Client, version 11.6.0 resolving cookbooks for run list: ["apache", "apache::vhosts"] Synchronizing Cookbooks: - apache Compiling Cookbooks... Converging 11 resources Recipe: apache::default * package[apache2] action install (up to date) * service[apache2] action start (up to date) * service[apache2] action enable (up to date) * execute[a2dissite default] action run (skipped due to only_if) Recipe: apache::vhosts * template[/etc/apache2/sites-available/bears] action create (up to date) * execute[a2ensite bears] action run (skipped due to not_if) * directory[/srv/apache/bears] action create (up to date) * template[/srv/apache/bears/index.html] action create (up to date) * template[/etc/apache2/sites-available/clowns] action create (up to date) * execute[a2ensite clowns] action run (skipped due to not_if) * directory[/srv/apache/clowns] action create (up to date) * template[/srv/apache/clowns/index.html] action create (up to date) Chef Client finished, 0 resources updated
193. Hands on Recap
194. Our cookbook... • Chef cookbook "apache" with two recipes: • default (manages apache package and service) • vhosts (iterates over data bags and renders vhost configuration) • However, Opscode publishes an "apache2" cookbook that manages much much more, including all apache2 configuration, modules, sites Debian style w/ a2ensite/a2enmod, etc
195. Our role • webserver role in JSON • Sets a default attribute • Sets a run list • Uploaded with knife upload
196. Our data bag • The vhosts data bag serves as an example • Directory structure follows the data bag API end point and "knife upload" makes it easy to upload everything • Create additional vhosts to see how this is dynamically expanded easily
197. Chef Server • Hosted Enterprise Chef • Free to manage up to five nodes
198. Using A Different Chef Server • Configure knife by modifying the chef_server_url and the node_name values in ./chef/knife.rb • Get the validation client key from the Chef Server (this differs by implementation): • Open Source: •/etc/chef-server/chef-validator.pem • Hosted Chef / Private Chef • Download after creating an organization
199. $ cat .chef/knife.rb The knife.rb configuration # See http://docs.opscode.com/config_rb_knife.html for more information on knife configuration options current_dir = File.dirname(__FILE__) log_level :info log_location STDOUT node_name "nathenharveyrmr1" client_key "#{current_dir}/nathenharveyrmr1.pem" validation_client_name "nhrmr1-validator" validation_key "#{current_dir}/nhrmr1-validator.pem" chef_server_url "https://api.opscode.com/organizations/nhrmr1" cache_type 'BasicFile' cache_options( :path => "#{ENV['HOME']}/.chef/checksums" ) cookbook_path ["#{current_dir}/../cookbooks"]
200. Chef Resources
201. Chef Resources (Core Chef) • Chef client comes with 24+ different resources • Packages, files, services, users, symlinks, registry keys, and more • Each resource has one or more providers • Some resources have platform-specific providers (e.g., package, service, user, group) http://docs.opscode.com/resource.html
202. Managing files and directories • file • cookbook_file • remote_file • template • directory • remote_directory • link http://docs.opscode.com/resource.html
203. Managing packages • package • apt_package • chef_gem • dpkg_package • easy_install_package • freebsd_package • gem_package • ips_package • macports_package • pacman_package • portage_package • rpm_package • smartos_package • solaris_package • yum_package http://docs.opscode.com/resource.html
204. RubyGem packages • chef_gem - install a RubyGem into Chef's Ruby environment to be used in a Chef recipe • gem_package - install a RubyGem to be used by the system or an application http://docs.opscode.com/resource.html
205. Services • service is used to manage services using the common init systems available • Each platform has it's own provider • arch, debian, freebsd, gentoo, "init", insserv • invokercd, macosx (launchd), redhat, "simple" • solaris (SMF), upstart, windows http://docs.opscode.com/resource.html
206. Other Chef Resources • cron • deploy (revision, timestamped) • env (windows) • group • ifconfig (RHEL) • log • mdadm • mount • ohai • registry_key (windows) • route • scm (git, subversion) • user http://docs.opscode.com/resource.html
207. There when you need them... • execute (we used this) • script (bash, perl, python, csh, ruby interpreters) • windows_script (batch, powershell) - Chef 11.6.0! • ruby_block http://docs.opscode.com/resource.html
208. Cookbooks to Know About
209. community.opscode.com • apache2, nginx • ark • build-essential • chef-client • chruby • cron • line • java • jenkins • minitest-handler • mysql, postgresql • openssh • omnibus_updater • partial_search • runit • whitelist-node-attrs
210. Opscode Cookbooks with Chef Resources • Opscode has several cookbooks that include new custom Chef Resources • apt (apt_repository) • aws (aws_ebs_volume, aws_elastic_ip, and more) • yum (yum_repository) • windows (windows_package, windows_feature, and more) • homebrew (homebrew_package) • runit (runit_service) • many more! http://docs.opscode.com/chef/lwrps_custom.html
211. Getting Cookbooks from the Community Site • Knife commands: •knife cookbook site download • knife cookbook site install (integrates with git) • Dependency resolvers (a la Ruby's bundler): •librarian-chef •berkshelf
212. Tools to be aware of...
213. Knife Plugins • http://docs.opscode.com/community_plugin_knife.html • Cloud plugins (ec2, openstack, rackspace, google azure, hpcloud, cloudstack, eucalyptus, and more) • knife-server • knife-solo • knife-preflight • knife-essentials • https://github.com/jkeiser/knife-essentials
214. Chef Handlers • http://docs.opscode.com/community_plugin_report_handler.html • IRC • Campfire • HipChat • DataDog • Splunk Storm • Graylog2 • Graphite
215. Workflow Helpers • Berkshelf: berkshelf.com • Librarian-chef: github.com/applicationsonline/ librarian-chef • Knife Spork: github.com/jonlives/knife-spork • Vagrant: vagrantup.com (also a testing tool)
216. Cookbook Testing • Vagrant: vagrantup.com • Test Kitchen: github.com/opscode/test-kitchen • Foodcritic: acrmp.github.io/foodcritic • ChefSpec: acrmp.github.io/chefspec/ • RSpec: rspec.info • minitest-chef: github.com/calavera/minitest-chef- handler • cookbook: minitest-handler
217. Thank you! Questions & Answers nharvey@opscode.com joshua@opscode.com