Upgrade to Pro — share decks privately, control downloads, hide ads and more …

Testing Infrastructure with Terratest

Testing Infrastructure with Terratest

How to Test Infrastructure Code using Terraform, Terratest from Gruntwork.io

Ladislav Prskavec

April 06, 2020
Tweet

More Decks by Ladislav Prskavec

Other Decks in Technology

Transcript

  1. Who Am I 4 Oracle Cloud Infrastructure - API Services

    team 4 Twitter: @abtris 4 Website: https://www.prskavec.net/ 4 Golang Prague organizer (@GoMeetupPrague) Ladislav Prskavec - Teststack conference, 4. 6. 2020 2
  2. Yevgeniy Brikman 4 co-founder of Gruntwork with Josh Padnick 4

    Gruntwork are authors of tools: Terragrunt and Terratest 4 Books Author - Terraform: Up & Running and Hello, Startup 4 Yevgeniy Brikman talk Automated Testing for Terraform, Docker, Packer, Kubernetes, and More, 11 Nov 2019 Ladislav Prskavec - Teststack conference, 4. 6. 2020 7
  3. Syntax validation 4 terraform validate 4 kubernetes apply -f <file>

    --dry-run -- validate=true Ladislav Prskavec - Teststack conference, 4. 6. 2020 11
  4. Dry-run 4 terraform plan 4 kubectl apply -f <file> --server-dry-run

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 12
  5. Linting & policies checks 4 hadolint 4 kube-score 4 conftest

    - Open Policy Agent Rego query language Ladislav Prskavec - Teststack conference, 4. 6. 2020 13
  6. Test strategy (per unit) 1. Deploy real infrastructure 2. Validate

    it works 3. Undeploy the infrastructure Ladislav Prskavec - Teststack conference, 4. 6. 2020 16
  7. Dockerfile FROM alpine RUN echo 'Hello, World!' > /test.txt Ladislav

    Prskavec - Teststack conference, 4. 6. 2020 18
  8. Test package test import ( "testing" "github.com/gruntwork-io/terratest/modules/docker" "github.com/magiconair/properties/assert" ) func

    TestDockerHelloWorldExample(t *testing.T) { tag := "teststack/demo-terratest-docker" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, "../hello-world", buildOptions) opts := &docker.RunOptions{Command: []string{"cat", "/test.txt"}} output := docker.Run(t, tag, opts) assert.Equal(t, "Hello, World!", output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 19
  9. Test strategy - integration test 1. Deploy all the infrastruture

    2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 23
  10. Basic example OCI - APIGW and Fn Ladislav Prskavec -

    Teststack conference, 4. 6. 2020 24
  11. const fdk = require('@fnproject/fdk'); fdk.handle(function(input) { let name = 'World';

    if (input.name) { name = input.name; } return {message: 'Hello ' + name}; }); Ladislav Prskavec - Teststack conference, 4. 6. 2020 25
  12. schema_version: 20180708 name: hello-world version: 0.0.2 runtime: node entrypoint: node

    func.js format: http-stream triggers: - name: hello-world-trigger type: http source: /hello-world-trigger Ladislav Prskavec - Teststack conference, 4. 6. 2020 26
  13. package test import (...) func TestHelloWorldAppUnit(t *testing.T) { uniqueID :=

    random.UniqueId() terraformOptions := &terraform.Options{ TerraformDir: "../", Vars: map[string]interface{}{ "name": fmt.Sprintf("hello-world-%s", uniqueID), "suffix": fmt.Sprintf("%s", uniqueID), "tenancy_ocid": os.Getenv("TF_VAR_tenancy_ocid"), "user_ocid": os.Getenv("TF_VAR_user_ocid"), "private_key_path": os.Getenv("TF_VAR_private_key_path"), "fingerprint": os.Getenv("TF_VAR_fingerprint"), "private_key_password": os.Getenv("TF_VAR_private_key_password"), "region": os.Getenv("TF_VAR_region"), }, } defer terraform.Destroy(t, terraformOptions) terraform.InitAndApply(t, terraformOptions) validateHelloWorldApp(t, terraformOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 27
  14. func validateHelloWorldApp(t *testing.T, terraformOptions *terraform.Options) { url := terraform.Output(t, terraformOptions,

    "gateway_url") expectedStatus := 200 expectedBody := `{"message":"Hello World"}` maxRetries := 10 timeBetweenRetries := 10 * time.Second http_helper.HttpGetWithRetry(t, url, nil, expectedStatus, expectedBody, maxRetries, timeBetweenRetries ) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 28
  15. Validate what make sense for you 4 test terraform output

    4 http calls api/application 4 tcp/udp calls 4 ssh commands Ladislav Prskavec - Teststack conference, 4. 6. 2020 33
  16. Parallelism 4 locking resources issue, 4 separations (compartments, accounts, regions)

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 35
  17. Resource conflicts 4 randomize 4 namespace (k8s) / compartments (OCI)

    / accounts (AWS) import "github.com/gruntwork-io/terratest/modules/random" randomSuffix := random.UniqueId() // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) Ladislav Prskavec - Teststack conference, 4. 6. 2020 37
  18. import test_structure "github.com/gruntwork-io/terratest/modules/test-structure" func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose"

    test_structure.RunTestStage(t, "build_docker_image", func() { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 39
  19. Why skip stages 4 skip cleanup on first run 4

    skip deploy/build/cleanup on iterations Ladislav Prskavec - Teststack conference, 4. 6. 2020 40
  20. How skip stages 4 use environment variable SKIP_stage_name=1 4 run

    with that: SKIP_build_docker_image=1 go test Ladislav Prskavec - Teststack conference, 4. 6. 2020 41
  21. Why and how make retries 4 Exponential Backoff And Jitter

    4 Simulator for AWS architecture blog (http://www.awsarchitectureblog.com/ ) about jitter and backoff. Ladislav Prskavec - Teststack conference, 4. 6. 2020 42
  22. How make retries import http_helper "github.com/gruntwork-io/terratest/modules/http-helper" retries := 30 sleepBetweenRetries

    := 5*time.Second http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", retries, sleepBetweenRetries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 43
  23. How make retries with curl opts := &docker.RunOptions{ Command: []string{"--retry",

    "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) Ladislav Prskavec - Teststack conference, 4. 6. 2020 44
  24. apiVersion: apps/v1 kind: Deployment metadata: name: hello-world-deployment spec: selector: matchLabels:

    app: hello-world replicas: 1 template: metadata: labels: app: hello-world spec: containers: - name: hello-world image: training/webapp:latest ports: - containerPort: 5000 Ladislav Prskavec - Teststack conference, 4. 6. 2020 46
  25. kind: Service apiVersion: v1 metadata: name: hello-world-service spec: selector: app:

    hello-world ports: - protocol: TCP targetPort: 5000 port: 5000 type: LoadBalancer Ladislav Prskavec - Teststack conference, 4. 6. 2020 47
  26. package test import (...) func TestKubernetesHelloWorldExample(t *testing.T) { kubeResourcePath :=

    "../hello-world-kubernetes/deployment.yaml" // unique namespace namespaceName := fmt.Sprintf("test-%s", strings.ToLower(random.UniqueId())) options := k8s.NewKubectlOptions("", "", namespaceName) k8s.CreateNamespace(t, options, namespaceName) defer k8s.DeleteNamespace(t, options, namespaceName) defer k8s.KubectlDelete(t, options, kubeResourcePath) k8s.KubectlApply(t, options, kubeResourcePath) k8s.WaitUntilServiceAvailable(t, options, "hello-world-service", 10, 3*time.Second) service := k8s.GetService(t, options, "hello-world-service") url := fmt.Sprintf("http://%s", k8s.GetServiceEndpoint(t, options, service, 5000)) http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello world!", 30, 5*time.Second) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 48
  27. 1. Build docker images for nginx and webapp 2. Run

    docker-compose 3. Validate nginx and webapp 4. Destroy docker-compose Ladislav Prskavec - Teststack conference, 4. 6. 2020 51
  28. version: "3" services: nginx: image: local/nginx:latest container_name: "production_nginx" networks: -

    "teststack-network" ports: - "80:80" go-webapp: image: go-webapp:latest container_name: "production_go-webapp" environment: - SERVER_TEXT=${SERVER_TEXT} expose: - "8080" networks: - "teststack-network" networks: "teststack-network": Ladislav Prskavec - Teststack conference, 4. 6. 2020 52
  29. package main import (...) func helloHandler(res http.ResponseWriter, req *http.Request) {

    res.Header().Set( "Content-Type", "text/plain", ) io.WriteString( res, fmt.Sprintf("%s", os.Getenv("SERVER_TEXT")), ) } func defaultHandler(w http.ResponseWriter, r *http.Request) { fmt.Fprintf(w, "Go web app powered by Docker") } func main() { http.HandleFunc("/", defaultHandler) http.HandleFunc("/hello", helloHandler) err := http.ListenAndServe(":8080", nil) if err != nil { log.Fatal("ListenAndServe: ", err) return } } Ladislav Prskavec - Teststack conference, 4. 6. 2020 53
  30. func TestDockerComposeLocal(t *testing.T) { tag := "go-webapp" buildargs := "-t

    local/nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) tag := "local/nginx" buildargs := "-f Dockerfile.nginx" buildOptions := &docker.BuildOptions{ Tags: []string{tag}, BuildArgs: []string{buildargs} } docker.Build(t, "../hello-world-docker-compose", buildOptions) Ladislav Prskavec - Teststack conference, 4. 6. 2020 54
  31. serverPort := 80 randomSuffix := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!",

    randomSuffix) dockerOptions := &docker.Options{ WorkingDir: "../hello-world-docker-compose", EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "SERVER_PORT": strconv.Itoa(serverPort), "randomSuffix": randomSuffix, }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") maxRetries := 10 timeBetweenRetries := 5 * time.Second url := fmt.Sprintf("http://localhost:%d/hello", serverPort) tlsConfig := tls.Config{} http_helper.HttpGetWithRetry(t, url, &tlsConfig, 200, expectedServerText, maxRetries, timeBetweenRetries) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 55
  32. func TestDockerComposeWithStagesLocal(t *testing.T) { workingDir := "../hello-world-docker-compose" test_structure.RunTestStage(t, "build_docker_image", func()

    { buildImage(t, "go-webapp", "../demowebapp") buildImage(t, "local/nginx", "../nginx") }) test_structure.RunTestStage(t, "run_docker_compose", func() { runCompose(t, workingDir) }) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 57
  33. func buildImage(t *testing.T, tag string, workingDir string) { buildOptions :=

    &docker.BuildOptions{ Tags: []string{tag}, } docker.Build(t, workingDir, buildOptions) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 58
  34. func runCompose(t *testing.T, workingDir string) { serverPort := 80 randomSuffix

    := random.UniqueId() expectedServerText := fmt.Sprintf("Hello, %s!", randomSuffix) dockerOptions := &docker.Options{ WorkingDir: workingDir, EnvVars: map[string]string{ "SERVER_TEXT": expectedServerText, "randomSuffix": randomSuffix, "SERVER_PORT": strconv.Itoa(serverPort), }, } defer docker.RunDockerCompose(t, dockerOptions, "down") docker.RunDockerCompose(t, dockerOptions, "up", "-d") opts := &docker.RunOptions{ Command: []string{"--retry", "5", "--retry-connrefused", "-s", "http://production_nginx:80/hello"}, OtherOptions: []string{"--network", "testdockercomposewithstageslocal_teststack-network"}, } tag := "appropriate/curl" output := docker.Run(t, tag, opts) assert.Equal(t, expectedServerText, output) } Ladislav Prskavec - Teststack conference, 4. 6. 2020 59
  35. Can we use this test strategy? 1. Deploy all the

    infrastruture 2. Validate it works 3. Undeploy all the infrastruture Ladislav Prskavec - Teststack conference, 4. 6. 2020 64
  36. Test type Number of resources Chance of failure Unit test

    10 1% Integration test 50 5% E2E test 500+ 40%+ For unit & integration you improve reliability with retries. Ladislav Prskavec - Teststack conference, 4. 6. 2020 65
  37. Please don't write E2E tests in this way. Ladislav Prskavec

    - Teststack conference, 4. 6. 2020 66
  38. Incremental testing 1. Deploy a persistent test environment and leave

    it running. 2. Each time you update a module, deploy & validate just that module. 3. Test your deployment process is zero-downtime too. Ladislav Prskavec - Teststack conference, 4. 6. 2020 68
  39. Summary 4 Static Analysis (1-60 seconds) 4 Unit Tests (1-20

    minutes) 4 Integration Tests (5-60 minutes) 4 E2E Tests (60-240+ minutes) Ladislav Prskavec - Teststack conference, 4. 6. 2020 69
  40. What Good Worse Static Analysis Very Fast (until 1 min),

    Stable, Easy to use, No deploy Limited errors for catch, no reals checks with infrastructure Unit tests Fast (1-20 min), very good stability with retries, trust in components Need real deploy, non trivial test code, extra work for CI Integration tests Mostly stable with retries, High level confidence in multiple units Slow (5-60 min), Need real deploy, non trivial test code, extra work for CI E2E tests Build confidence in entire architecture Very slow (60-240+ min), Need real deploy, Still problem 0- deployments, Can be brittle (even with retries) Ladislav Prskavec - Teststack conference, 4. 6. 2020 70
  41. Try make at least one test for your docker environment!

    Ladislav Prskavec - Teststack conference, 4. 6. 2020 71