Test-Driven Development (TDD) for Infrastructure

Originally presented at 2019 O'Reilly Velocity (Berlin).

In software development, test-driven development (TDD) is the process of writing tests and then developing functionality to pass the tests. Let's explore methods of adapting and applying TDD to configuring and deploying infrastructure-as-code.

Repository here: https://github.com/joatmon08/tdd-infrastructure

Transcript

1. Copyright © 2019 HashiCorp Test-Driven Development for Infrastructure Rosemary Wang, @joatmon08 Velocity Berlin | Nov. 7, 2019
2. “TDD is dead.” https://dhh.dk/2014/tdd-is-dead-long-live-testing.html
3. What’s Test- Driven Development (TDD)? ▪ Write tests firsts, then the feature. ▪ “Red, green, refactor” ▪ Benefits: – Only build defined features – Modular, testable code
4. Integration Tests Contract Tests Unit Tests “Ideal” Testing Pyramid Manual Testing Cost (Time, $$$) End-to-End Tests
5. Infrastructure Testing Signpost End-to-End Tests Manual Testing Integration Tests Contract Tests Unit Tests Cost (Time, $$$)
6. Why is it a signpost? ▪ #YOLO-driven development ▪ Lack of education ▪ Lack of automation / interface ▪ Lack of tooling
7. Let’s try TDD for Infrastructure. ▪ Infrastructure configuration still requires good code practices ▪ Infrastructure architecture has definition of “feature complete” ▪ terraform yolo isn’t a thing (or insert deploy command here).
8. Target State
9. Integration Tests Contract Tests Unit Tests TDD Up the Pyramid Manual Testing Cost (Time, $$$) End-to-End Tests
10. Warning, Some Terraform VS Code Live Share at: hashi.co/tdd-velocity
11. ⁄ Unit Testing
12. Unit Tests ▪ Test individual component ▪ Can be run without dependencies ▪ Infrastructure: check specific configuration & syntax
13. Live Code ▪ We’ll use conftest (Open Policy Agent) & terraform validate. ▪ Terraform-specific tools, compiled by Peter Souter: – clarity – terraform-compliance – terraform_validate ▪ Other infrastructure: native testing frameworks
14. Target State
15. Unit TDD Function logic? Execution? Resource dependencies? Fast feedback Architectural Conformance Lint and check syntax
16. ⁄ Contract Testing
17. Contract Tests ▪ Validate interactions between 2 components (input & output) ▪ “Real” resources not required ▪ Compare infrastructure state
18. Live Code ▪ We’ll use conftest (Open Policy Agent) & terraform plan. ▪ Terraform-specific tools: kitchen-terraform ▪ Can also use native testing frameworks – Example for network switches with Python – Example for S3 Bucket Policy with Golang – If Kubernetes, check Kubernetes to external components (i.e., DNS)
19. Target State
20. Contract TDD Resource dependencies? Interactions? Fast feedback Check functional logic Catch our input errors More holistic
21. ⁄ Integration Testing
22. Integration Tests ▪ Confirms interactions between 2+ components ▪ Real resources required ▪ May also include acceptance tests
23. IMPORTANT ▪ Integration tests are about interactions. ▪ When using IaC tool, testing deployment is redundant. – In Terraform, covered by Provider Acceptance Tests ▪ Many types of integration tests, including Functional, Policy, & Security
24. Testing with Kitchen
25. Tools ▪ Terraform-specific tools: terratest, kitchen-terraform, Sentinel ▪ Other tools: – AWS localstack / mocking framework (USE WITH CAUTION) – Kubernetes local environments (KIND, Minikube, etc.) – Various Inspec resource packs
26. Fails! TERMINAL > kitchen test -----> Starting Kitchen (v2.3.3) … Waiting for SSH service on 54.93.35.169:22, retrying in 3 seconds Waiting for SSH service on 54.93.35.169:22, retrying in 3 seconds Waiting for SSH service on 54.93.35.169:22, retrying in 3 seconds Waiting for SSH service on 54.93.35.169:22, retrying in 3 seconds Waiting for SSH service on 54.93.35.169:22, retrying in 3 seconds
27. Passing the Test
28. Pass! TERMINAL > kitchen test -----> Starting Kitchen (v2.3.3) … Profile: Integration Tests for Application (default) ✔ db: Database: check routing from public to private subnet ✔ Host 10.128.0.43 port 27017 proto tcp should be reachable ✔ Host 10.128.0.43 port 27017 proto tcp should be resolvable ✔ Host 10.128.0.43 port 80 proto tcp should not be reachable ✔ outbound: Public Subnet: check routing out to public internet ✔ HTTP GET on https://hashicorp.com status should cmp == 301 Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped Test Summary: 3 successful, 0 failures, 0 skipped
29. Integration TDD Not that fast Requires real resources Convert theory to reality Caught a missing component Isolate sections of system
30. ⁄ End-to-End Testing
31. End-to-End Tests ▪ “The Real Deal” ▪ Can be manual, fully automated, or smoke tests ▪ If it starts smoking, it doesn’t work.
32. TDD it! Did it fail?
33. Tools ▪ Terraform-specific tools: terratest, Sentinel ▪ Other: – Behavior-Driven Development (BDD) Frameworks – Can also use native testing frameworks – Kubernetes: Sonobuoy conformance tests, local environments (KIND, Minikube, etc.)
34. ⁄ Final Thoughts
35. Ugh, so much effort. ▪ (Yes, it can be.) ▪ Unit testing is useless here! ▪ I don’t find testing tools for this! ▪ Stop with pedantic practices!
36. [Preaching] TDD is dead. So is #YOLO-driven development.
37. Use TDD to Learn Infrastructure ▪ Confidence in infrastructure-as-code ▪ Develop knowledge of infrastructure ▪ More useful integration & end-to-end tests ▪ Tacit knowledge of a change’s blast radius
38. Eventually, YDD & TDD of infrastructure are dead …to you.
39. joatmon08.github.io/#slides Please rate the session! Rosemary Wang she/her Developer Advocate at HashiCorp rosemary@hashicorp.com @joatmon08 joatmon08 linkedin.com/in/rosemarywang/ 39