Testing IaC — How we could do it
Hi guys, how are you? I hope you’re all doing fine.
In my previous article we discussed about a Terraform code and why we should run tests for our infrastructure.
Now I’m going to show how I did, so maybe you do for your own code, so we could all learn together.
There is a repository for this article, which is right here. You can check it if you have any doubts or even to be inspired or if you wish to make a PR. Go for it.
Learning about Terratest
First of all, let’s cover the tool necessary to run the tests. Created by Gruntwork, Terratest is a Golang library that enables us to create unit testing for Terraform.
I really recommend you to read the documentation and check out some examples for you to be familiar with the coding.
The first thing you note is that you must have some Golang basic knowledge. If you want to learn it, go check this course if you want a PT-BR version or this one if you want a EN-US version.
When you are ready to begin, make sure to install Terratest, they have it covered in the documentation, and make sure you Go is well configured.
Understanding the Basics of Terratest
Terratest has a nice way of testing Terraform. It’ll automatically create, test and destroy your infrastructure. So let’s analyze each part we have in our code.
func TestTerraformAws(t *testing.T){ t.Parallel()
terraformOptions := &terraform.Options{ TerraformDir: "../../terraform/", Vars: map[string]interface{}{ "region": awsRegion, }, }}defer terraform.Destroy(t, terraformOptions)terraform.InitAndApply(t, terraformOptions)
This is the foundation of Terratest.
First of all, we set our test to be parallel, so if we have multiple functions, they could all run in parallel. But knowing it is Terraform, and we’d create and destroy resources in the cloud, this could take a while. Near the end we’ll se a way to avoid this.
Then, we create a variable containing options for our Terraform code. This will be used. We need two components, the directory of our Terraform code and the variables which are going to be set during the apply. We can pass any variable in this configuration as long as it is declared in our Terraform code.
After this, there is a line which will destroy the Terraform, and you might be wondering “Why does it comes before the InitAndApply? Shouldn’t it be the other way around?” No, because in Golang we have this defer which informs that a function can only be executed after all the other functions in the same scope.
Finally there is the InitAndApply which will download the necessary modules and apply your code.
For all of our tests, we will set the following variable:
awsRegion := aws.GetRandomStableRegion(t, nil, nil)This will return a random and valid AWS Region, which then will be passed to our test, so we can see if it works in any AWS Region.
I’d also like to add that the easier(and maybe only) way to test, is by capturing the Terraform Outputs. Terratest has a nice form of doing this, like this:
vpcID := terraform.Output(t, terraformOptions, "main_vpc_id")But I need to tell you, this is not optimal. Because Terraform can output strings, arrays, and maps. But Terratest only sees strings. You are going to see this during this article, but we are primarily dealing with strings here. So pay attention during the course to this section. I didn’t. Be advised.
Testing our Network
In the Terraform code that we are testing, I told you we created two subnets, one private and another public. Now, for our test, we want something simple to test, like if the public subnet is really public, and if the private subnet is really private.
For AWS a public subnet is one that has a 0.0.0.0/0 route pointing to a Internet Gateway. If not, it is a private subnet.
So we are testing three things:
- we are creating the right number of subnets
- the public subnets are really public
- the private subnets are really privates
Now, we need to collect the information we need, which are:
- The VPC Id
- Subnets Ids
We do it like this:
publicSubnetIDs := terraform.Output(t, terraformOptions, "public_subnets_id")privateSubnetIDs := terraform.Output(t, terraformOptions, "private_subnets_id")vpcID := terraform.Output(t, terraformOptions, "main_vpc_id")
As I said previously, Terratest only sees strings, so we need to format our output to match our needs, specially in the subnets, because we need an array in order to test them all.
replacer := strings.NewReplacer("[", "", "]", "", "\"", "", "\n", "", " ", "")subnetPublID := replacer.Replace(publicSubnetIDs)arrayPublSubnets := strings.Split(subnetPublID, ",")
subnetPrivID := replacer.Replace(privateSubnetIDs)arrayPrivSubnets := strings.Split(subnetPrivID, ",")
The replacer is a special use of a library called strings, we need it for removing all the special characters in the output. Then we split the strings into an array.
Now we test it, first, let’s see if we have the right amount of subnets:
require.Equal(t, 4, len(aws.GetSubnetsForVpc(t, vpcID, awsRegion)))Great! Now we test the subnets:
for i := 0; i < len(arrayPrivSubnets)-1; i++ { isPrivate, err := aws.IsPublicSubnetE(t, arrayPrivSubnets[i], awsRegion) if err != nil { fmt.Printf("Error Encountered: %s", err) return } assert.False(t, isPrivate)}for i := 0; i < len(arrayPublSubnets)-1; i++ { isPublic, err := aws.IsPublicSubnetE(t, arrayPublSubnets[i], awsRegion) if err != nil { fmt.Printf("Error Encountered: %s", err) return } assert.True(t, isPublic)}
Great, and just like this we could test our VPC.
Testing our RDS
Now for our RDS we need to grab two values:
- the RDS Instance Id
- The connection string stored in AWS SSM Parameter Store
We could grab the Id and the Name of the Parameter with Terratest:
RDSInstanceID := terraform.Output(t, terraformOptions, "rds_id")connectionStringParameter := terraform.Output(t, terraformOptions, "rds_connection_string_parameter")
And now we can get to work!
First, we get the value of the encrypted parameter:
key, err := aws.GetParameterE(t, awsRegion, connectionStringParameter)if err != nil { fmt.Printf("Error Encountered in getting AWS Parameter: %s\n", err) return}
This value is a string, we need to convert it to a JSON, for to work with it, we could do it like this:
connToJSON := []byte(key)var JSONMapConnString map[string]interface{}if err := json.Unmarshal(connToJSON, &JSONMapConnString); err != nil { fmt.Printf("Error Encountered in Unmarshal: %s\n", err) return}
Now we could get the information we need:
expectedPort := int64(3306)expectedDatabaseName := fmt.Sprint(JSONMapConnString["DATABASE"])username := fmt.Sprint(JSONMapConnString["USER"])password := fmt.Sprint(JSONMapConnString["PASS"])
There are more values in this JSON, but I didn’t want to get them this way, cause I want to test some Golang functions.
That’s why we get them using these:
address, err := aws.GetAddressOfRdsInstanceE(t, RDSInstanceID, awsRegion)if err != nil { fmt.Printf("Error Encountered in getting RDS Address: %s\n", err) return}port, err := aws.GetPortOfRdsInstanceE(t, RDSInstanceID, awsRegion) if err != nil { fmt.Printf("Error Encountered in getting RDS Port: %s\n", err)return}schemaExistsInRdsInstance, err := aws.GetWhetherSchemaExistsInRdsMySqlInstanceE(t, address, port, username, password, expectedDatabaseName)if err != nil { fmt.Printf("Error Encountered in getting RDS Schema: %s\n", err) return}
When we have those values, we can test them:
assert.NotNil(t, address)assert.Equal(t, expectedPort, port)assert.True(t, schemaExistsInRdsInstance)
We are testing if the address of the database is not null, if the port is same as expected above and if there is a database as we created in Terraform.
Testing our SSM
For testing our SSM Connection, we need only one information: the instance Id. But, as you saw, we are using an AutoScaling Group, so we can't get the instances information from our module, so we need to get a little creative here. I searched the Internet on how to do this in an easy way and I found out about Terraform's Data External. It is basically a script you could run an get the output as long as it's in JSON format. So I did it this way:
data "external" "instance_id" {
program = ["bash", "${path.root}/scripts/getInstanceIDFromAsg.sh"] query = {
region = var.region
asg_name = module.asg.this_autoscaling_group_name }
depends_on = [module.asg, module.db]
}
This will work fine in Linux, because we have access to bash in our terminal, for Windows users I left a Python version in the same folder.
The script is basically doing a get in the AWS Api and returning all the instances related to an AutoScaling Group:
function get_instances_ids() {
RESULT=$(aws autoscaling describe-auto-scaling-groups \
--auto-scaling-group-names ${ASG_NAME} \
--region ${REGION} \
--query AutoScalingGroups[].Instances[].InstanceId \
--output json)
jq -n --arg v "$RESULT" '{"instances_id": $v}'
}With this information in hand, we can now output it and get it with Terratest.
instanceOutput := terraform.Output(t, terraformOptions, "instances_ip")replacer := strings.NewReplacer("[", "", "]", "", "\"", "", "\n", "", " ", "", "{", "", "}", "") instancesIds := replacer.Replace(instanceOutput)
arrayInstances := strings.Split(instancesIds, ",")
Next step is to wait for the instance to be available:
for i := 0; i < len(arrayInstances); i++ {
aws.WaitForSsmInstance(t, awsRegion, arrayInstances[i], timeout)
}And then we can test some command response:
for i := 0; i < len(arrayInstances); i++ {
result, err := aws.CheckSsmCommandE(t, awsRegion, arrayInstances[i], "echo Hello, World", timeout) fmt.Printf("Checking instance %s", arrayInstances[i])
if err != nil {
fmt.Printf("Error Encountered in checking SSM Stdout: %s\n", err)
return
}
assert.Equal(t, result.Stdout, "Hello, World\n")
assert.Equal(t, result.Stderr, "")
assert.Equal(t, int64(0), result.ExitCode) }
With this, you tested if the SSM Agent is installed, has the correct Policies and if we could access it.
Testing our application
Now, for the most exciting part, we need to test if our application is responding what we expect. If you remember the previous article I said that our application returns:
- the region which was deployed
- an unique Id
- the database version
All we need for this last test is the ALB DNS which we can get pretty easily like this:
albDNS := terraform.Output(t, terraformOptions, "alb-dns")Now we just create a JSON to match the result, the one we expect to get:
httpJSON := map[string]interface{}{
"database_version": fmt.Sprintf("%s-log", mysqlVersion),
"region": awsRegion,
"unique_id": uniqueID,
}expectedResult, _ := json.Marshal(httpJSON)
And then we use the http_helper library to send the request:
http_helper.HttpGetWithRetry(t, URL, nil, 200, string(expectedResult), maxRetries, timeBetweenRetries)With this, we are testing if our ALB is up and running, if our app has connectivity to our database and if we have the right policies.
Terratest Test Structure
If you have come so far, you may be wondering: "Ok, this is cool, but Am I going to be creating and destroying my infrastructure all the time just for one test?"
For sure you won't. That's why we have the test_structure library. It enables us to create Stages in our test, so we don't have to create and destroy multiple times. I've run into some problems with it, mostly because I didn't read the documentation properly(I know, I'm dumb) but I'm here so you don't make the same mistakes I made.
I've created three stages: Deploy, Validate, Cleanup.
Look at the basic structure of a stage call:
test_structure.RunTestStage(t, "deploy", func() {
test_structure.SaveString(t, workingDir, savedAwsRegion, awsRegion)
deployUsingTerraform(t, awsRegion, workingDir, uniqueID)
})We have three arguments. The first one is the test we are running, the second one is the name of the stage. And the last one is a group of functions related to that stage.
The main thing you have to notice is that you need to save and load your Terraform Option in each function called. For example:
func validateRds(t *testing.T, workingDir string, awsRegion string) {
deploy_terraform stage terraformOptions := test_structure.LoadTerraformOptions(t, workingDir) RDSInstanceID := terraform.Output(t, terraformOptions, "rds_id")...
This will make sure you are getting the correct resources. So, for this to work, first you need to save your option during the Deploy Stage, like this:
terraformOptions := &terraform.Options{
TerraformDir: workingDir,
Vars: map[string]interface{}{
"region": awsRegion,
"unique_id": uniqueID, },
} test_structure.SaveTerraformOptions(t, workingDir, terraformOptions)
All the other parts follow the same structure we had so far. Just have this little thing, of saving the Terraform Options and loading in mind, it will help you a lot.
Guys, I really want you to enjoy this article, find it helpful and maybe you could try a little bit of this. I promise you it's fun at the end.
If you had any doubts, you can send me a DM in my LinkedIn and we could learn together.
See ya!
