Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Introduction To Continuous Compliance & Remedia...

Nathen Harvey
September 07, 2017

Introduction To Continuous Compliance & Remediation

Success with DevOps can be measured with a number of different metrics. How frequently are systems audited for compliance to various policies? How long does it take to remediate a failing control or vulnerability? This workshop provide an introduction to practice of continuous compliance and remediation. The workshop uses InSpec and Chef for compliance and remediation, respectively. InSpec is an open-source testing framework for infrastructure with a human-readable language for specifying compliance, security and other policy requirements. Chef is an open-source framework for infrastructure automation. Easily integrate automated tests that check for adherence to policy into any stage of your deployment pipeline.

Nathen Harvey

September 07, 2017
Tweet

More Decks by Nathen Harvey

Other Decks in Technology

Transcript

  1. 55% Step one: Detect Gain visibility into current status to

    satisfy audits and drive decision-making of organizations do compliance assessments inconsistently or not at all. Apply policies and gain a complete view across the fleet ▪  Accurately assess risk ▪  Prioritize remediation actions ▪  Maintain audit readiness ▪  Create and adjust policies ” Continuous visibility means that you enter into audits knowing the outcome. Jon Williams, NIU ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓
  2. ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓

    ✓ ✓ ✓ Step two: Correct Remediate issues to improve performance and security ▪  Prioritize actions based on impact ▪  Improve application performance ▪  Close security holes ▪  Prove policy compliance Web & Media Giant Can patch 250,000 nodes within 6 hours of a patch being made available Develop, test, and deploy remediation to address issues across the fleet ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ of organizations need days or longer to remediate issues. 58%
  3. Today's Workshop •  Detect a compliance failure with InSpec and

    Chef Automate •  Create a Chef cookbook to remediate the failure •  Test the cookbook with Test Kitchen •  Remediate the failure with the new cookbook •  Validate our remediation in Chef Automate
  4. •  Login to Chef Automate •  Find your workstation/node • 

    Find your workstation’s IP address •  Login to your workstation Access the Learning Environment
  5. Let's log in to Chef Automate! •  https://34.214.86.220 •  Uses

    a self-signed certificate in this lab •  Username: chef •  Password: chef
  6. $ Log in to your remote workstation ssh [email protected] -p

    443 The authenticity of host 12.34.56.78 (12.34.56.78)' can't be established. ECDSA key fingerprint is SHA256:zAtoeO29XbhRNvwg542cuh4qsKCEaX8hNIlEOCbgd3I. Are you sure you want to continue connecting (yes/no)?
  7. $ Log in to your remote workstation ssh [email protected] -p

    443 The authenticity of host 12.34.56.78 (12.34.56.78)' can't be established. ECDSA key fingerprint is SHA256:zAtoeO29XbhRNvwg542cuh4qsKCEaX8hNIlEOCbgd3I. Are you sure you want to continue connecting (yes/no)? yes
  8. $ Log in to your remote workstation ssh [email protected] -p

    443 The authenticity of host 12.34.56.78 (12.34.56.78)' can't be established. ECDSA key fingerprint is SHA256:zAtoeO29XbhRNvwg542cuh4qsKCEaX8hNIlEOCbgd3I. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '12.34.56.78' (ECDSA) to the list of known hosts. [email protected]'s password:
  9. $ Log in to your remote workstation ssh [email protected] -p

    443 The authenticity of host 12.34.56.78 (12.34.56.78)' can't be established. ECDSA key fingerprint is SHA256:zAtoeO29XbhRNvwg542cuh4qsKCEaX8hNIlEOCbgd3I. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added '12.34.56.78' (ECDSA) to the list of known hosts. [email protected]'s password: update_me
  10. $ List your home directory ls -t nathen-harvey cookbooks Berksfile

    profiles nodes Berksfile.lock config.json
  11. $ Verify the installation chef --version Chef Development Kit Version:

    2.0.26 chef-client version: 13.2.20 delivery version: master (17c1b0fed9be4c70f69091a6d21a4cbf0df60a23) berks version: 6.2.0 kitchen version: 1.16.0 inspec version: 1.32.1
  12. Chef DK - The Chef Development Kit Foodcritic Test Your

    "Chef Style" •  Validate your Chef code against Chef best practices •  Extend with rules to enforce organizational Chef development best practices •  Enforce compliance & security practices CookStyle Validate your Ruby •  Validate your Chef code against Ruby best practices •  Identify potential Ruby errors (unclosed strings, etc.) •  Identify style/convention that helps write better code (single quotes vs. double quotes) ChefSpec Simulate Chef •  Validate your Chef code will run •  Testing for more Chef advanced used cases •  Useful for regression testing Test Kitchen Let's do this (almost) for real •  Validate your Chef code against Chef best practices •  Extend with rules to enforce organizational Chef development best practices •  Enforce compliance & security practices InSpec Verify automation results & ensure compliance •  Assert the intention of your Chef code •  Verify on live systems that your Chef code produced the correct result •  Confirm your Chef code did not produce compliance drift or failures
  13. $ Run chef run_chef [2017-03-10T14:05:49+00:00] INFO: Forking chef instance to

    converge... Starting Chef Client, version 12.18.31 ... Converging 0 resources [2017-03-10T14:05:51+00:00] INFO: Chef Run complete in 0.19413018 seconds Running handlers: [2017-03-10T14:05:51+00:00] INFO: Running report handlers Running handlers complete [2017-03-10T14:05:51+00:00] INFO: Report handlers complete Chef Client finished, 0/0 resources updated in 01 seconds
  14. $ Run Chef with the audit cookbook run_chef "recipe[audit::default]" [2017-03-10T14:10:34+00:00]

    INFO: Forking chef instance to converge... Starting Chef Client, version 12.18.31 [2017-03-10T14:10:34+00:00] INFO: *** Chef 12.18.31 *** ... [2017-03-10T14:10:40+00:00] INFO: Chef Run complete in 4.10402964 seconds Running handlers: [2017-03-10T14:10:40+00:00] INFO: Running report handlers [2017-03-10T14:10:40+00:00] WARN: Format is json [2017-03-10T14:10:40+00:00] INFO: Initialize InSpec [2017-03-10T14:10:40+00:00] INFO: Running tests from: [{:name=>"ssh", :path=>"/home/chef/profiles/ssh"}] [2017-03-10T14:10:40+00:00] INFO: Reporting to chef-automate ... Running handlers complete [2017-03-10T14:10:40+00:00] INFO: Report handlers complete Chef Client finished, 1/2 resources updated in 06 seconds
  15. Audit Cookbook •  Installs InSpec (if necessary - included in

    Chef 13 by default) •  Run InSpec profiles •  Report results to Chef Automate
  16. •  Allows for functionality added in Chef 12.5 to be

    used in Chef 12.1 or later •  Includes: •  custom resource functionality •  notification improvements •  new resources added to Chef The audit cookbook uses it to ensure as many customers can use it as possible, but it should be avoided in Chef 13 and later. Compat Resource Cookbook
  17. $ cat config.json Attributes for the Audit cookbook { "audit":

    { "collector": "chef-automate", "profiles": [ { "name": "ssh", "path": "/home/chef/profiles/ssh" } ] } }
  18. $ Our ssh InSpec profile tree profiles/ssh ssh ├── controls

    │ └── ssh.rb ├── inspec.lock └── inspec.yml 2 directories, 3 files
  19. $ Our ssh InSpec profile cat profiles/ssh/controls/ssh.rb control 'sshd-1.0' do

    impact 0.7 title 'SSH Version 2' desc 'Only SSH version 2 should be enabled' describe sshd_config do its('Protocol') { should cmp 2 } end end
  20. $ Run locally with InSpec inspec exec profiles/ssh Profile: SSH

    Configuration (ssh) Version: 0.1.0 Target: local:// × sshd-1.0: SSH Version 2 ( expected: 2 got: (compared using `cmp` matcher) ) × SSH Configuration Protocol should cmp == 2 expected: 2 got: (compared using `cmp` matcher) Profile Summary: 0 successful, 1 failures, 0 skipped Test Summary: 0 successful, 1 failures, 0 skipped
  21. Next Steps •  Automate the remediation of the failing control

    •  Test the remediation before deploying •  Deploy the remediation, and use the audit cookbook to report back to Automate •  View the compliant node in Automate
  22. •  A recipe to deploy a proper sshd_config configuration file

    •  A local test environment configured to test our changes Create an SSH Chef Cookbook
  23. $ Generate a new ssh cookbook chef generate cookbook ssh

    Generating cookbook ssh - Ensuring correct cookbook file content - Committing cookbook files to git - Ensuring delivery configuration - Ensuring correct delivery build cookbook content - Adding delivery configuration to feature branch - Adding build cookbook to feature branch - Merging delivery content feature branch to master Your cookbook is ready. Type `cd ssh` to enter it. There are several commands you can run to get started locally developing and testing your cookbook. Type `delivery local --help` to see a full list. Why not start by writing a test? Tests for the default recipe are stored at: test/smoke/default/default_test.rb If you'd prefer to dive right in, the default recipe can be found at: recipes/default.rb
  24. $ Add a server recipe to the ssh cookbook chef

    generate recipe ssh server Recipe: code_generator::recipe * directory[./ssh/spec/unit/recipes] action create (up to date) * cookbook_file[./ssh/spec/spec_helper.rb] action create_if_missing (up to date) * template[./ssh/spec/unit/recipes/server_spec.rb] action create_if_missing - create new file ./ssh/spec/unit/recipes/server_spec.rb - update content in file ./ssh/spec/unit/recipes/server_spec.rb from none to d14960 (diff output suppressed by config) * directory[./ssh/test/smoke/default] action create (up to date) * template[./ssh/test/smoke/default/server.rb] action create_if_missing - create new file ./ssh/test/smoke/default/server.rb - update content in file ./ssh/test/smoke/default/server.rb from none to aa8bba (diff output suppressed by config) * template[./ssh/recipes/server.rb] action create - create new file ./ssh/recipes/server.rb - update content in file ./ssh/recipes/server.rb from none to 18f24e (diff output suppressed by config)
  25. $ Add a template to the cookbook chef generate template

    ssh sshd_config -s /etc/ssh/sshd_config Recipe: code_generator::template * directory[./ssh/templates/default] action create - create new directory ./ssh/templates/default * file[./ssh/templates/sshd_config.erb] action create - create new file ./ssh/templates/sshd_config.erb - update content in file ./ssh/templates/sshd_config.erb from none to a16b11 (diff output suppressed by config)
  26. Remember... Infrastructure policies need testing! •  Linting •  Static analysis

    •  Unit testing •  Integration Testing •  Compliance Testing "Infrastructure as Code" should be tested like ANY other codebase.
  27. Test-Driven Development •  Write a test, watch it fail • 

    Write some code •  Write and run more tests •  Code review •  Delivery pipeline to production •  Lowered chance of production failure
  28. Test Kitchen Configuration (2 of 3) ~/cookbooks/ssh/.kitchen.yml + - -

    ... platforms: - name: ubuntu-16.04 - name: centos-7.2 - name: centos-7.3 ...
  29. Test Kitchen Configuration (3 of 3) ~/cookbooks/ssh/.kitchen.yml + - +

    - + - suites: - name: default - name: server run_list: - recipe[ssh::default] - recipe[ssh::server] verifier: inspec_tests: - test/smoke/default - /home/chef/profiles/ssh attributes:
  30. $ List the kitchens kitchen list Instance Driver Provisioner Verifier

    Transport Last Action Last Error server-centos-73 Docker ChefZero Inspec Ssh <Not Created> <None>
  31. $ Converge kitchen converge -----> Starting Kitchen (v1.15.0) ... ----->

    Creating <server-centos-73>... Sending build context to Docker daemon 227.8 kB Sending build context to Docker daemon Step 0 : FROM centos:centos7 ... Running handlers: [2017-03-12T02:26:16+00:00] INFO: Running report handlers Running handlers complete [2017-03-12T02:26:16+00:00] INFO: Report handlers complete Chef Client finished, 1/1 resources updated in 01 seconds Finished converging <server-centos-73> (0m23.54s). -----> Kitchen is finished. (1m0.39s)
  32. $ Verify the Kitchen kitchen verify -----> Verifying <server-centos-73>... Loaded

    Target: ssh://kitchen@localhost:32771 × sshd-1.0: SSH Version 2 ( expected: 2 got: (compared using `cmp` matcher) ) × SSH Configuration Protocol should cmp == 2 expected: 2 got: (compared using `cmp` matcher) Profile Summary: 0 successful, 1 failures, 0 skipped Test Summary: 0 successful, 1 failures, 0 skipped
  33. Edit the SSH Configuration Template ~/cookbooks/ssh/templates/sshd_config.erb - + #ListenAddress 0.0.0.0

    #ListenAddress :: # The default requires explicit activation of protocol 1 #Protocol 2 Protocol 2 # HostKey for protocol version 1
  34. $ Converge (apply our new cookbook change) kitchen converge ----->

    Starting Kitchen (v1.15.0) ... -----> Converging <server-centos-73>... ... # The default requires explicit activation of protocol 1 -#Protocol 2 +Protocol 2 # HostKey for protocol version 1 ... Running handlers: [2017-03-12T02:32:32+00:00] INFO: Running report handlers Running handlers complete [2017-03-12T02:32:32+00:00] INFO: Report handlers complete Chef Client finished, 1/1 resources updated in 01 seconds Finished converging <server-centos-73> (0m16.32s). -----> Kitchen is finished. (0m17.34s)
  35. $ Verify the Kitchen kitchen verify -----> Starting Kitchen (v1.15.0)

    ... -----> Verifying <server-centos-73>... Loaded Target: ssh://kitchen@localhost:32771 ✔ sshd-1.0: SSH Version 2 ✔ SSH Configuration Protocol should cmp == 2 Profile Summary: 1 successful, 0 failures, 0 skipped Test Summary: 1 successful, 0 failures, 0 skipped Finished verifying <server-centos-73> (0m0.22s). -----> Kitchen is finished. (0m1.27s)
  36. $ End-to-End Kitchen Test kitchen test -----> Starting Kitchen (v1.15.0)

    ... -----> Cleaning up any prior instances of <server-centos-73> -----> Destroying <server-centos-73>... ... -----> Testing <server-centos-73> -----> Creating <server-centos-73>... ... -----> Creating <server-centos-73>... ... Finished creating <server-centos-73> (0m0.60s). -----> Converging <server-centos-73>... ...
  37. $ End-to-End Kitchen Test kitchen test -----> Installing Chef Omnibus

    (install only if missing) ... -----> Setting up <server-centos-73>... Finished setting up <server-centos-73> (0m0.00s). -----> Verifying <server-centos-73>... ... Profile Summary: 1 successful, 0 failures, 0 skipped Test Summary: 1 successful, 0 failures, 0 skipped Finished verifying <server-centos-73> (0m0.51s). -----> Destroying <server-centos-73>... ... -----> Kitchen is finished. (0m25.18s)
  38. $ Remediate with Chef run_chef "recipe[ssh::server],recipe[audit::default]" [2017-03-10T16:48:02+00:00] INFO: Forking chef

    instance to converge... Starting Chef Client, version 12.18.31 ... Synchronizing Cookbooks: - ssh (0.1.0) - audit (2.4.0) - compat_resource (12.16.3) ... -#Protocol 2 +Protocol 2 ... [2017-03-10T16:48:05+00:00] INFO: Chef Run complete in 1.248588588 seconds Running handlers: ... [2017-03-10T16:48:05+00:00] INFO: Report handlers complete Chef Client finished, 1/3 resources updated in 03 seconds
  39. Get started with •  https://learn.chef.io/modules/chef-automate-pilot/ Set up your own demo

    environment •  https://downloads.chef.io/automate Install on-prem, generate a trial license •  AWS OpsWorks for Chef Automate Managed service •  AWS and Azure Marketplace