Unit Testing Your Kubernetes Configuration with Open Policy Agent

Talk from KubeCon EU 2019, focused on using Open Policy Agent to write tests for structured configuration.

Transcript

1. Using Open Policy Agent Gareth Rushgrove | Director, Product Management | Snyk snyk.io Unit testing your Kubernetes configuration May, 2019
2. snyk.io
3. snyk.io - A quick introduction to Open Policy Agent - Shift-left testing - Introducing conftest - Rego as a programming language - Portability between different Kubernetes solutions - Not just Kubernetes Agenda
4. Open Policy Agent snyk.io A policy enforcement engine for configuration
5. What is Open Policy Agent?
6. Policy Data OPA Service How Open Policy Agent works
7. A growing ecosystem
9. Shift left snyk.io Faster feedback
10. Shift-left testing is an approach to software testing and system testing in which testing is performed earlier in the lifecycle (i.e., moved left on the project timeline). Wikipedia
11. Development cycle Fast Slow Slower Cluster Local development Continuous integration
12. Open Policy Agent is normally used here Development cycle Cluster Local development Continuous integration
13. Cluster What if we could use Open Policy Agent here as well? Development cycle Local development Continuous integration
14. “Everything as code” Said several times at the CDF event
15. “Would you write code without tests?”
16. Introducing conftest snyk.io Test your configuration locally, and in CI
17. Introducing conftest
18. apiVersion: apps/v1 kind: Deployment metadata: name: hello-kubernetes spec: replicas: 3 selector: matchLabels: app: hello-kubernetes template: metadata: labels: app: hello-kubernetes spec: containers: - name: hello-kubernetes image: paulbouwer/hello-kubernetes:1.5 ports: Given a deployment snyk.io
19. package main deny[msg] { input.kind = "Deployment" not input.spec.template.spec.securityContext.runAsNonRoot = true msg = "Containers must not run as root" } deny[msg] { input.kind = "Deployment" not input.spec.selector.matchLabels.app msg = "Containers must provide app label for pod selectors" } Write your policies snyk.io
20. deny[msg] { input.kind = "Deployment" not input.spec.template.spec.securityContext.runAsNonRoot = true msg = "Containers must not run as root" } Explaining what we just wrote snyk.io We should deny any input for which Deployment is the value for kind and When runAsNonRoot is set to false
21. $ conftest test deployment.yaml deployment.yaml Containers must not run as root $ echo $status 1 Run tests locally with conftest snyk.io
22. snyk.io Demo
23. Rego as a language snyk.io The usual pros and cons of a DSL
24. Good documentation
25. The Rego playground
26. package main test_deployment_without_security_context { deny["Containers must not run as root"] with input as {"kind": "Deployment"} } test_deployment_with_security_context { no_violations with input as {"kind": "Deployment", "spec": { "selector": { "matchLabels": { "app": "something", "release": "something" }}, "template": { "spec": { "securityContext": { "runAsNonRoot": true }}}}} } test_services_not_denied { no_violations with input as {"kind": "Service"} } test_services_issue_warning { warn["Services are not allowed"] with input as {"kind": "Service"} Built-in testing tools
27. $ opa test --verbose . data.main.test_deployment_without_security_context: PASS (1.029µs) data.main.test_deployment_with_security_context: PASS (1.058µs) data.main.test_services_not_denied: PASS (701ns) data.main.test_services_issue_warning: PASS (614ns) --------------------------------------------------------------------- PASS: 4/4 Open Policy Agent test runner snyk.io
28. In:path .rego extension:rego 546 results Not much public rego code yet
29. Open Policy Agent Bundles
30. package main # has_field returns whether an object has a field has_field(object, field) { object[field] } # False is a tricky special case, as false responses would create an undefined # document unless they are explicitly tested for has_field(object, field) { object[field] == false } has_field(object, field) = false { General helpers
31. package main empty(value) { count(value) == 0 } no_violations { empty(deny) } no_warnings { empty(warn) } Test helpers snyk.io
32. package kubernetes is_service { input.kind = "Service" } is_deployment { input.kind = "Deployment" } Domain specific helpers snyk.io
33. Reusing OCI registries
34. Proposed OCI media types of OPA
35. $ ls kubernetes.rego $ conftest push instrumenta.azurecr.io/kubernetes-helpers $ ls policy $ conftest pull instrumenta.azurecr.io/kubernetes-helpers:latest $ ls policy kubernetes.rego Using conftest to share policy snyk.io
36. Bundle Bar
37. Portability snyk.io Helping to move between different Kubernetes tools
38. snyk.io Demo
39. $ kustomize build | conftest test - Kustomize snyk.io
40. $ helm template | conftest test - Helm snyk.io
41. import {Pod} from 'kubernetes-types/core/v1' import {ObjectMeta} from 'kubernetes-types/meta/v1' import * as yaml from 'js-yaml' let metadata: ObjectMeta = {name: 'example', labels: {}} let pod: Pod = { apiVersion: 'v1', kind: 'Pod', metadata, spec: { containers: [ /* ... */ ], }, } console.log(yaml.safeDump(pod)) Typescript
42. $ npx ts-node pod.ts | conftest test - Typescript snyk.io
43. $ kubectl get all -o json \ | jq -cj '.items[] | tostring+"\u0000"' \ | xargs -n1 -0 -I@ bash -c "echo '@' | conftest test -" Kubectl (look away now) snyk.io
44. $ kubectl krew install conftest $ kubectl conftest deployment some-deployment Kubectl plugin snyk.io
45. package kubernetes deployment "hello-kubernetes": { apiVersion: "apps/v1" spec: { replicas: 3 template spec containers: [{ image: "paulbouwer/hello-kubernetes:1.5" ports: [{ containerPort: 8080 }] }] } } Cue snyk.io
46. package kubernetes import "encoding/yaml" command test: { task conftest: { kind: "exec" cmd: "conftest test -" stdin: yaml.MarshalStream(objects) } } Cue snyk.io
47. $ cue test Containers must not run as root --- . command "conftest test -" failed: exit status 1 terminating because of errors Cue snyk.io
48. Not just Kubernetes snyk.io Lots of other configurations to care about
49. apiVersion: kind: extension:yaml 1,054,453 results Lots of Kubernetes-like docs
50. in:path .yml language:YAML 69,822,306 results Lots more YAML
51. snyk.io Demo
52. $ terraform plan -out config.tfplan $ terraform show -json config.tfplan | configtest test - Terraform snyk.io
53. package main blacklist = [ "google_iam", "google_container" ] deny[msg] { check_resources(input.resource_changes, blacklist) banned := concat(", ", blacklist) msg = sprintf("Terraform plan will change prohibited resources in: %v", [banned]) } # Checks whether the plan will cause resources with certain prefixes to change check_resources(resources, disallowed_prefixes) { startswith(resources[_].type, disallowed_prefixes[_]) } Terraform snyk.io
54. service: aws-python-scheduled-cron frameworkVersion: ">=1.2.0 <2.0.0" provider: name: aws runtime: python2.7 tags: author: "this field is required" functions: cron: handler: handler.run runtime: python2.7 events: - schedule: cron(0/2 * ? * MON-FRI *) Serverless framework snyk.io
55. package main deny[msg] { input.provider.runtime = "python2.7" msg = "Python 2.7 cannot be the default provider runtime" } runtime[name] { input.functions[i].runtime = name } deny[msg] { runtime["python2.7"] msg = "Python 2.7 cannot be used as the runtime for functions" } deny[msg] { not has_field(input.provider.tags, "author") msg = "Should set provider tags for author" } Serverless framework snyk.io
56. version: "3.4" services: web: build: . ports: - "5000:5000" redis: image: "redis:latest" Docker Compose snyk.io
57. package main version { to_number(input.version) } deny[msg] { endswith(input.services[_].image, ":latest") msg = "No images tagged latest" } deny[msg] { version < 3.5 msg = "Must be using at least version 3.5 of the Compose file format" } Docker Compose snyk.io
58. Conclusions snyk.io If all you remember is...
59. @jdolitsky @stevelasker @sometorin Thanks open source community!
60. snyk.io - Open Policy Agent is incredibly flexible - Check out conftest at github.com/instrumenta/conftest - Expect lots more integrations in the future - Managing configuration as code needs better tools Come talk to Snyk at booth #S41 Summary
61. Questions? snyk.io And thanks for listening