Beyond terraform apply

As infrastructure-as-code (IaC) becomes the backbone of modern cloud deployments, ensuring the reliability and correctness of our Terraform configurations is more critical than ever. In this comprehensive guide, we’ll explore a range of testing strategies, from community best practices to Terraform’s native testing capabilities, to help you build rock-solid infrastructure code.

Community Insights on Terraform Testing

Recently, our community of practice session shed light on various approaches to Terraform testing. Let’s recap some key strategies before diving into Terraform’s native testing features.

Validation and Formatting

The most basic form of “testing” involves simply validating the syntax and formatting of our Terraform files. Tools like terraform validate and terraform fmt provide quick checks to ensure our code is syntactically correct and follows consistent formatting conventions.

Linting

Going beyond basic validation, linting tools can help enforce best practices and catch potential issues early. TFLint was highlighted as a popular option, capable of identifying deprecated syntax, unused variables, and security risks.

An example .tflint.hcl configuration file would look like this:

plugin "aws" {  
  enabled = true  
  version = "0.21.1"  
  source  = "github.com/terraform-linters/tflint-ruleset-aws"  
}  
  
rule "terraform_deprecated_index" {  
  enabled = true  
}  
  
rule "terraform_unused_declarations" {  
  enabled = true  
}  
  
rule "terraform_comment_syntax" {  
  enabled = true  
}

Unit Testing

While traditional unit testing may seem less applicable to declarative code, tools like Terratest allow you to write Go code to validate the behavior of your Terraform modules.

Using terratest to confirm that a module has the right output would look like this:

package test  
  
import (  
    "testing"  
    "github.com/gruntwork-io/terratest/modules/terraform"  
    "github.com/stretchr/testify/assert"  
)  
  
func TestTerraformHelloWorldExample(t *testing.T) {  
    terraformOptions := &terraform.Options{  
        TerraformDir: "../",  
    }  
  
    defer terraform.Destroy(t, terraformOptions)  
  
    terraform.InitAndApply(t, terraformOptions)  
  
    output := terraform.Output(t, terraformOptions, "hello_world")  
    assert.Equal(t, "Hello, World!", output)  
}

Integration Testing

Integration testing becomes crucial when working with multiple interconnected resources. Terratest can be used to spin up real infrastructure, verify its properties, and then tear it down — all within an automated test suite.

A more complex example of using Terratest to verify that a terraform config looks as expected would look like this:

package test  
  
import (  
    "testing"  
    "github.com/gruntwork-io/terratest/modules/terraform"  
    "github.com/gruntwork-io/terratest/modules/aws"  
    "github.com/stretchr/testify/assert"  
)  
  
func TestAWSInfrastructure(t *testing.T) {  
    t.Parallel()  
  
    terraformOptions := &terraform.Options{  
        TerraformDir: "../",  
        Vars: map[string]interface{}{  
            "region": "us-west-2",  
        },  
    }  
  
    defer terraform.Destroy(t, terraformOptions)  
  
    terraform.InitAndApply(t, terraformOptions)  
  
    vpcID := terraform.Output(t, terraformOptions, "vpc_id")  
    vpc := aws.GetVpcById(t, vpcID, "us-west-2")  
  
    assert.Equal(t, "10.0.0.0/16", vpc.CidrBlock)  
}

Compliance Testing

Ensuring your infrastructure meets security and compliance standards is critical. Tools like Checkov can scan your Terraform code for potential security misconfigurations and compliance violations.

Checkov is simple to configure, you can create a .checkov.yaml file in your project root to customise checks:

skip-check:  
  - CKV_AWS_18  # Skip check for default VPC usage  
  - CKV_AWS_23  # Skip check for encryption of S3 bucket  
check:  
  - ALL  # Run all other checks

End-to-End (E2E) Testing

E2E testing involves validating entire infrastructure stacks in a production-like environment. This can catch issues that might only surface when multiple components interact.

E2E testing often involves setting up a complete infrastructure and running tests against it. Here’s a simplified example using Terratest:

package test  
  
import (  
    "testing"  
    "github.com/gruntwork-io/terratest/modules/terraform"  
    "github.com/gruntwork-io/terratest/modules/http-helper"  
)  
  
func TestEndToEndInfrastructure(t *testing.T) {  
    t.Parallel()  
  
    terraformOptions := &terraform.Options{  
        TerraformDir: "../",  
    }  
  
    defer terraform.Destroy(t, terraformOptions)  
  
    terraform.InitAndApply(t, terraformOptions)  
  
    url := terraform.Output(t, terraformOptions, "application_url")  
  
    http_helper.HttpGetWithRetry(t, url, nil, 200, "Hello, World!", 30, 5)  
}

Drift Detection

Infrastructure drift occurs when the actual state of your cloud resources diverges from what’s defined in your Terraform code. Regular drift detection, either through terraform plan or specialised tools like driftctl, helps catch unauthorised changes and keep your infrastructure in sync with your code.

For drift detection, you can use terraform plan in your CI/CD pipeline:

terraform plan -detailed-exitcode

This command will exit with:

• 0 if there are no changes
• 1 if there’s an error
• 2 if there are changes

You can also use driftctl scanfor more advanced drift detection, which can be configured via a .driftignore file:

# Ignore specific resource  
aws_iam_user.example  
# Ignore all resources of a type  
aws_iam_*

Policy Simulation

Using cloud provider policy simulators (like the handy AWS IAM policy simulator) to test access controls defined in Terraform.

data "aws_caller_identity" "current" {}  
  
data "aws_iam_principal_policy_simulation" "s3_object_access" {  
  action_names = [  
    "s3:GetObject",  
    "s3:PutObject",  
    "s3:DeleteObject",  
  ]  
  policy_source_arn = data.aws_caller_identity.current.arn  
  resource_arns     = ["arn:aws:s3:::my-test-bucket"]  
  
  # The "lifecycle" and "postcondition" block types are part of  
  # the main Terraform language, not part of this data source.  
  lifecycle {  
    postcondition {  
      condition     = self.all_allowed  
      error_message = <<EOT  
        Given AWS credentials do not have sufficient access to manage ${join(", ", self.resource_arns)}.  
      EOT  
    }  
  }  
}

Contract Testing

Verifying that modules are used correctly by checking input variables against expected formats and values.

For contract testing, we’ll create a Terraform module with input variable validation and then test it using terraform test as a taster for the next section.

First, create a module with input validation:

# modules/example/main.tfvariable "instance_type" {  
  type        = string  
  description = "EC2 instance type"  validation {  
    condition     = can(regex("^t[23].(micro|small|medium)$", var.instance_type))  
    error_message = "Instance type must be t2 or t3 micro, small, or medium."  
  }  
}resource "aws_instance" "example" {  
  instance_type = var.instance_type  
  ami           = "ami-12345678" # Replace with a valid AMI ID  
}output "instance_id" {  
  value = aws_instance.example.id  
}

Now, create a test file for this module:

# tests/example_test.tftest.hclvariables {  
  instance_type = "t2.micro"  
}run "valid_instance_type" {  
  command = plan  assert {  
    condition     = aws_instance.example.instance_type == var.instance_type  
    error_message = "Instance type does not match input"  
  }  
}run "invalid_instance_type" {  
  variables {  
    instance_type = "t2.large"  
  }  command = plan  expect_failures = [  
    {  
      summary_contains = "Instance type must be t2 or t3 micro, small, or medium."  
    }  
  ]  
}

To run these tests, simply run terraform test in your terminal or in CI.

Cost Testing (trust me, that’s a thing)

Using tools like infracost you can easily get an estimate for how much the infrastructure that you have currently costs you — with caveats around usage-based pricing — and how much your new infrastructure would cost.

Infracrost in action. Source: https://github.com/infracost/infracost/tree/master

This allows you to review pull requests based on the cost difference introduced by the change, and an opportunity to ask questions if the difference is higher than expected.

Terraform’s Native Testing Capabilities

While third-party tools offer robust testing options, Terraform itself has introduced powerful native testing capabilities with the terraform test command. Let's explore how you can leverage this built-in functionality to create automated tests for your Terraform modules and configurations.

Test File Structure

Terraform discovers test files by looking for the .tftest.hcl or .tftest.json extensions. These files use a structure similar to regular Terraform configurations but with special blocks for defining tests.

# example.tftest.hcl  
  
variables {  
  region = "us-west-2"  
}  
  
provider "aws" {  
  region = var.region  
}  
  
run "example_test" {  
  command = plan  
}

Run Blocks

The core of a Terraform test file is the run block. Each run block represents a sequence of Terraform operations (like plan or apply) along with assertions about the expected outcomes.

run "test_s3_bucket_creation" {  
  command = plan  
  
  variables {  
    bucket_name = "my-test-bucket"  
  }  
  
  assert {  
    condition     = length(plan.resource_changes) > 0  
    error_message = "No changes were detected in the plan."  
  }  
  
  assert {  
    condition     = plan.resource_changes[0].type == "aws_s3_bucket"  
    error_message = "Expected an S3 bucket to be created."  
  }  
}

Variables and Providers

Test files can include variables and provider blocks, allowing you to set up the necessary context for your tests.

variables {  
  region = "us-west-2"  
  environment = "test"  
}  
  
provider "aws" {  
  region = var.region  
}  
  
run "test_with_variables" {  
  command = plan  
  
  variables {  
    instance_type = "t2.micro"  
  }  
  
  assert {  
    condition     = aws_instance.example[0].instance_type == "t2.micro"  
    error_message = "Instance type does not match expected value."  
  }  
}

Assertions

Within run blocks, you can define multiple assert blocks to check various conditions about the plan or applied state.

run "test_vpc_configuration" {  
  command = plan  
  
  assert {  
    condition     = aws_vpc.main.cidr_block == "10.0.0.0/16"  
    error_message = "VPC CIDR block is not as expected."  
  }  
  
  assert {  
    condition     = length(aws_subnet.public) == 3  
    error_message = "Expected 3 public subnets."  
  }  
  
  assert {  
    condition     = alltrue([for subnet in aws_subnet.public : subnet.map_public_ip_on_launch])  
    error_message = "All public subnets should have map_public_ip_on_launch set to true."  
  }  
}

Mock Providers

Terraform 1.7 introduced the ability to mock providers in tests, allowing you to test configurations without actually interacting with cloud APIs.

mock_provider "aws" {  
  mock_resource "aws_instance" {  
    defaults = {  
      instance_type = "t2.micro"  
      ami           = "ami-12345678"  
    }  
  }  
}  
  
run "test_with_mock_provider" {  
  command = plan  
  
  assert {  
    condition     = aws_instance.example.instance_type == "t2.micro"  
    error_message = "Instance type does not match expected value."  
  }  
  
  assert {  
    condition     = aws_instance.example.ami == "ami-12345678"  
    error_message = "AMI does not match expected value."  
  }  
}

Writing Effective Terraform Tests

Let’s explore some strategies for writing comprehensive tests using terraform test.

Test Module Inputs

Verify that your module behaves correctly with different input variables.

run "test_module_with_small_instance" {  
  command = plan  
  
  variables {  
    instance_type = "t2.small"  
  }  
  
  assert {  
    condition     = module.ec2_instance.instance_type == "t2.small"  
    error_message = "Instance type does not match input variable."  
  }  
}  
  
run "test_module_with_medium_instance" {  
  command = plan  
  
  variables {  
    instance_type = "t2.medium"  
  }  
  
  assert {  
    condition     = module.ec2_instance.instance_type == "t2.medium"  
    error_message = "Instance type does not match input variable."  
  }  
}

Validate Resource Configurations

Ensure that resources are configured as expected.

run "test_s3_bucket_configuration" {  
  command = plan  
  
  assert {  
    condition     = aws_s3_bucket.example.versioning[0].enabled  
    error_message = "S3 bucket versioning should be enabled."  
  }  
  
  assert {  
    condition     = aws_s3_bucket.example.server_side_encryption_configuration[0].rule[0].apply_server_side_encryption_by_default[0].sse_algorithm == "AES256"  
    error_message = "S3 bucket should use AES256 encryption."  
  }  
}

Check for Specific Plan Changes

Verify that your configuration makes the expected changes.

run "test_plan_changes" {  
  command = plan  
  
  assert {  
    condition     = length([for rc in plan.resource_changes : rc if rc.type == "aws_instance" && rc.change.actions[0] == "create"]) == 1  
    error_message = "Expected exactly one EC2 instance to be created."  
  }  
  
  assert {  
    condition     = length([for rc in plan.resource_changes : rc if rc.type == "aws_security_group" && rc.change.actions[0] == "create"]) == 1  
    error_message = "Expected exactly one security group to be created."  
  }  
}

Test Error Conditions

Ensure that your module fails gracefully with invalid inputs.

run "test_invalid_instance_type" {  
  command = plan  
  
  variables {  
    instance_type = "t2.invalid"  
  }  
  
  expect_failures = [  
    {  
      summary_contains = "Invalid instance type"  
    }  
  ]  
}

Best Practices for Terraform Testing

Before we end this article, here is a list of general best practices.

• Test edge cases. Don’t just test the happy path. Include tests for boundary conditions and error scenarios.
• Use mock providers. Leverage mock providers to test your logic without needing real cloud credentials or resources.
• Organise tests logically. Group related tests in the same file and use clear, descriptive names for your run blocks.
• Test module outputs. If your module produces outputs, write tests to verify their correctness.
• Include compliance checks. Use assertions to enforce tagging policies, naming conventions, or other organisational standards.
• Integrate with CI/CD. Run terraform test as part of your continuous integration pipeline to catch issues early.

Limitations and Considerations

While terraform test is powerful, it's important to understand its limitations:

• It’s primarily designed for testing modules rather than full configurations.
• Complex scenarios might still require external testing tools like Terratest.
• Mocking capabilities, while improving, may not cover all use cases.

Implementing a Comprehensive Testing Strategy

The key to effective Terraform testing is to combine multiple approaches. Start with basic validation and linting, then incrementally add unit tests for critical modules, integration tests for key resource interactions, and finally, end-to-end tests for critical infrastructure stacks.

Remember that testing infrastructure-as-code is an investment. While it may seem time-consuming initially, a robust testing strategy will save countless hours of debugging and prevent costly misconfigurations in production environments.

Conclusion

By leveraging both community-driven testing strategies and Terraform’s native testing capabilities, you can dramatically improve the reliability and maintainability of your infrastructure code. The terraform test command provides a powerful, built-in way to implement automated testing, while third-party tools offer additional capabilities for comprehensive coverage.

As you build your testing strategy, remember that the goal is not just to catch errors, but to give you confidence in your infrastructure changes. Start small, build up your test suite gradually, and enjoy the peace of mind that comes with well-tested Terraform code.

I encourage you to experiment with these techniques, share your experiences, and continue the discussion in the broader Terraform community. Happy testing, and here’s to more reliable, secure, and efficient infrastructure deployments!