Primer to automatic Azure infrastructure testing with Terratest
Intro
I recently had the opportunity to find a solution on automatic testing for Terraform-made infrastructure on Azure on a project I was working on.
The project’s setup was the following:
- Microsoft Azure, as a cloud platform
- Azure Repos, for version control
- Azure Pipelines (configured with yaml), for CICD
- Terraform, for IaC
After consulting Mr. Goole on what tools would be the best for testing the infra we were left with two candidates: Chef Inspec and Terratest. Terratest utilizes Go’s native testing functionality and has support for testing AWS, GCP, Azure, Docker, Packer, Terraform and Kubernetes. Terratest is more coding than writing templates, so our team saw an opportunity for growth there and chose Terratest.
Very soon I noticed that there was no good “getting started with Azure, Azure repos, Azure Pipelines, Terraform and Terratest”-guide to be found, so here is my contribution to the cause.
Table of contents
- Prerequisites
- Note on Go modules
- Writing a simple test
- Expanding the test
- Azure Pipelines
- What we achieved?
Prerequisites
On your machine, have
And also the following accounts:
- Azure (see below)
- Azure DevOps (see below)
- Git-based version control system (for Azure Repos, see below)
Getting an Azure & Azure DevOps account
You can easily get an Azure account with some usage credits and also gain access to all the free services even after the trial period ends. For Azure DevOps you can get a free account for organizations of five people or smaller, and it also comes with Azure Repos.
Note on Go modules
First hurdle for a Go newcomer like me was that if you develop Go, you have to do it in $GOPATH, which is where Go is installed on your system. But then there is this magical thing called “Go Modules” (from version 1.13 >=).
Basically on your /test-path you can just do
go mod init <verygoodname>
where it’s good karma to have verygoodname as your git repo address, if you are going to publish your module at some point. The only requirement seems to be that it should be locally unique, so just name it whatever.
Writing a simple test
Let’s get do the following basic example from Terratest’s home page, which should end up looking like this:
.
├── terraform
│ └── main.tf
└── test
└── simple_test.go
with the main.tf containing the following:
output "hello_world" { value = "Hello, World!" }
and simple_test.go:
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestTerraformHelloWorldExample(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../terratest",
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
output := terraform.Output(t, terraformOptions, "hello_world")
assert.Equal(t, "Hello, World!", output)
}
You can check from Terratest home page the more detailed description, but basically it initializes & applies the Terraform, runs the test(s) and in the end, destroys the infrastructure. The test itself is Terraform printing out “hello_world” and the Terratest checking whether the output is the same.
Running the test
In your /test-path first init your module (if you didn’t already)
go mod init <verygoodname>
and then run your test
go test
which runs all the _test-files in the path (there being just one) and giving us …
...
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Apply complete! Resources: 0 added, 0 changed, 0 destroyed.
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66:
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Outputs:
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66:
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: hello_world = Hello, World!
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 retry.go:72: terraform [output -no-color hello_world]
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Running command terraform with args [output -no-color hello_world]
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Hello, World!
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 retry.go:72: terraform [destroy -auto-approve -input=false -lock=false]
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66:
TestTerraformHelloWorldExample 2020-06-07T15:23:43+03:00 logger.go:66: Destroy complete! Resources: 0 destroyed.
PASS
ok gitlab.com/webziz/terratest.git 0.126s
… success!
That wasn’t so hard, wasn’t it?
Expanding the test
Let’s expand the test a little bit, and add:
- Support for Terraform variables and
- piece of infrastructure to be created in Azure
Terraform variables
Terraform projects (besides mundane examples) probably have some variables set to it. Let’s also use the opportunity to see a failing test.
Let’s create our envs for dev, and make it look like this:
.
├── envs
│ └── dev
│ └── variables.tfvars
├── terraform
│ ├── main.tf
│ └── vars.tf
└── test
├── go.mod
├── go.sum
└── simple_test.go
for variables.tfvars:
yell = "more, more, more"
and for vars.tfvars:
variable "yell" {
description = "Something worth yelling"
}
let’s also modify our main.tf:
output "hello_world" {
value = var.yell
}
and also modify terraformOptions in simple_test.go, so terraform gets the var-file as a parameter (exactly like terraform --vars varfile):
...
terraformOptions := &terraform.Options{
TerraformDir: "../terraform",
VarFiles: []string{
"../envs/dev/variables.tfvars",
},
}
...
Running the test
Now as we run the tests, it should fail miserably:
go test
...
--- FAIL: TestTerraformHelloWorldExample (0.20s)
simple_test.go:21:
Error Trace: simple_test.go:21
Error: Not equal:
expected: "Hello, World!"
actual : "more, more, more"
Diff:
--- Expected
+++ Actual
@@ -1 +1 @@
-Hello, World!
+more, more, more
Test: TestTerraformHelloWorldExample
FAIL
exit status 1
We failed, just like we planned to!
Azure infrastructure
For our test case our Azure infrastructure will be just a resource group.
Let’s add some files so our setup would look like this:
.
├── envs
│ └── dev
│ └── variables.tfvars
├── terraform
│ ├── main.tf
│ ├── output.tf
│ ├── resource_group.tf
│ └── vars.tf
└── test
├── go.mod
├── go.sum
└── simple_test.go
a new file resource_group.tf defining the new resource group:
resource "azurerm_resource_group" "target_rg" {
name = var.rg_name
location = var.location
}
let’s also go a bit OCD and add a new file output.tf to put all the outputs into:
output "hello_world" {
value = var.yell
}
output "rg_location" {
value = azurerm_resource_group.target_rg.location
}
leaving our main.tf with just the new azure provider information:
provider "azurerm" {
features {} # after version 2.0 this is mandatory
}
relevant variables added to vars.tf:
variable "yell" {
description = "Something worth yelling"
}
variable "location" {
description = "Azure location for resources"
}
variable "rg_name" {
description = "Resource Group name"
}
and also variables.tfvars:
yell = "more, more, more"
location = "westeurope"
rg_name = "test_rg"
Now, let’s add a stupid test and check whether our new resource group has the name we decided on it. Let’s also split the test into sub-test for better outputs:
package test
import (
"testing"
"github.com/gruntwork-io/terratest/modules/terraform"
"github.com/stretchr/testify/assert"
)
func TestSimpleTerraform(t *testing.T) {
terraformOptions := &terraform.Options{
TerraformDir: "../terraform",
VarFiles: []string{
"../envs/dev/variables.tfvars",
},
}
defer terraform.Destroy(t, terraformOptions)
terraform.InitAndApply(t, terraformOptions)
t.Run("Helloworld", func(t *testing.T) {
test1_output := terraform.Output(t, terraformOptions, "hello_world")
assert.Equal(t, "more, more, more", test1_output)
})
t.Run("Resource group location", func(t *testing.T) {
test2_output := terraform.Output(t, terraformOptions, "rg_location")
assert.Equal(t, "westeurope", test2_output)
})
}
You can find more info on the inner workings of tests from here.
Running the test
Before running any actual tests, we have to login into our Azure account:
az login
and after successful login we can once again run the test. Let’s try -v for more verbosity this time
go test -v
...
TestSimpleTerraformTest 2020-06-09T08:08:39+03:00 logger.go:66:
TestSimpleTerraformTest 2020-06-09T08:08:39+03:00 logger.go:66: Destroy complete! Resources: 1 destroyed.
--- PASS: TestSimpleTerraformTest (86.06s)
--- PASS: TestSimpleTerraformTest/Helloworld (0.04s)
--- PASS: TestSimpleTerraformTest/Resource_group_location (0.02s)
PASS
‘Yay’ for subtest verbosity, ‘boo’ for unimaginative test cases!
Azure Pipelines
If you want to continue following this “tutorial” by yourself, all the stuff has to be in version control to work (note: Gitlab does not work well with Azure Pipelines). So do that, or just use mine at github.
Start by flexing your fingers a bit, because setting up Azure Pipelines means some serious clicking around in the web console.
Start the actual process by creating a new (empty) azure_pipelines.yml file on the repo root, which should now look like this:
.
├── azure_pipelines.yml
├── envs
│ └── dev
│ └── variables.tfvars
├── terraform
│ ├── main.tf
│ ├── output.tf
│ ├── resource_group.tf
│ └── vars.tf
└── test
└── simple_test.go
Go to Azure DevOps portal and create a new Azure Pipeline, with your repository as a source. Choose to go for YAML and set the azure_pipelines.yml as your yml-file.
Set up authentication between Azure Pipelines and Azure
Now you should have access to the following:
ARM_CLIENT_ID: check from the main page of your new app registration in Azure Ad
ARM_TENANT_ID: check from the main page of your new app registration in Azure Ad
ARM_CLIENT_SECRET: the secret you got by following the HowTo
ARM_SUBSCRIPTION_ID: the subscription you are doing the infrastructure to. Check ‘subscriptions’
Go to your Pipelines in Azure DevOps and choose ‘Library’. Create a new variable group and give it a fancy name like ‘terraform-app-registration’ and store the above environmental variables with their actual values to the group.
Setting up Terraform
In order to get Terraform working in Azure Pipelines, you first have to install the add-on to your Azure DevOps account.
The actual pipeline yaml
For Azure pipelines I like to use native tasks when possible. They provide nice shortcuts when normally there would be only pain and misery.
In the variables-section we define the variable group we have put all our Terraform required authentication keys, that are magically usable by the pipeline tasks.
The pipeline definition should be relatively easy to read, assuming that you are familiar with YAML.
trigger:
- master
variables:
- group: terraform-app-registration
pool:
vmImage: 'ubuntu-latest'
steps:
- task: GoTool@0
displayName: "Install Go tooling"
inputs:
version: '1.13.5'
- task: TerraformInstaller@0
displayName: "Install Terraform tooling"
inputs:
terraformVersion: '0.12.3'
- bash: |
cd test
go mod init workaround
go test -v
displayName: "Run the tests"
And yes, there is bash, and it looks a little hacky, but I couldn’t get the GoTool@0 test-function to get the required packages, so with a workaround we go.
And if we look at the pipeline run …
TestSimpleTerraform 2020-06-10T04:54:13Z logger.go:66: azurerm_resource_group.target_rg: Destruction complete after 45s
TestSimpleTerraform 2020-06-10T04:54:13Z logger.go:66:
TestSimpleTerraform 2020-06-10T04:54:13Z logger.go:66: Destroy complete! Resources: 1 destroyed.
--- PASS: TestSimpleTerraform (70.06s)
--- PASS: TestSimpleTerraform/Helloworld (0.02s)
--- PASS: TestSimpleTerraform/Resource_group_location (0.02s)
PASS
ok workaround 70.067s
Finishing: Run bash script
It seems to work.
We now have a Terraform project creating Azure resources that is tested in Terratest automatically on commits to version control. Mission accomplished.
What we achieved?
Needless to say (again) that the test cases were bad. What I tried here to show with the really simple examples was having the whole shanigan Terraform - Terratest - Azure - Azure Pipelines all bundled together. Possibly in the process encouraging others to try the tools, as they are (relatively) easy it is to set up.
In Real Life™️ I wouldn’t do test cases like check from Terraform outputs whether the actual created resource matches the input vars. Terraform is built to ensure those things happen.
Some valid tests come into mind, like:
- “I have a policy in place, can I create resources that do not adhere to the policy” or
- “Does the web server respond after initialized” or
- “I just launched Terraform X, Y and Z, I wonder if my things A, B, C are still intact?”
For this blog post I wanted to keep things nice and simple, as the post length got totally out of hand even with the basics.