In my previous article, I provided a simple example of mocking an AWS resource using localstack, and testing with the python terraform-compliance module. In this example, I will provide a more extensive example using kitchen-terraform and terraform-compliance to deploy the following resources in AWS us-east-1 and us-west-2 regions.
- VPC
- Subnet
- Internet Gateway
- Route Table
- Route Table Association
- Security Group
- Key Pair
- 2 X EC2 Instance
To begin this example, you will need the following:
- Terraform
- Ruby
- Python3
- Python3 virtualenv module
- An AWS account with credentials configured in ~/.aws
- An AWS role or user with at least the minimum permissions:
{
"Version": "2012-10-17",
"Statement":
[
{
"Sid": "Stmt1469773655000",
"Effect": "Allow",
"Action": ["ec2:*"],
"Resource": ["*"]
}
]
}
Next, we need to set up a Python3 virtual environment, activate the environment and install the python terraform-compliance module.
which python3
/Library/Frameworks/Python.framework/Versions/3.8/bin/python3
cd ~
mkdir virtualenvs
cd virtualenvs
virtualenv terraform-test -p /Library/Frameworks/Python.framework/Versions/3.8/bin/python3
source terraform-test/bin/activate
pip install terraform-compliance
Now, we need to create a projects directory and download the sample code from github.
cd ~
mkdir projects
cd projects
git clone git@github.com:rubelw/terraform-kitchen.git
cd terraform-kitchen
Now we are ready to run our tests, by executing the ‘execute_kitchen_terraform.sh’ file.
This script will perform the following functions:
1. Install bundler
2. Install required gems
3. Create public and private key pair
4. Initialize terraform project
5. Test terraform plan output against terraform-compliance features
6. Execute kitchen test suite
- kitchen destroy centos(us-east-1)
- kitchen create centos(us-east-1)
- kitchen converge centos(us-east-1)
- kitchen verify centos (us-east-1)
- kitchen destroy centos(us-east-1)
- kitchen destroy ubuntu(us-west-2)
- kitchen create ubuntu(us-west-2)
- kitchen converge ubuntu(us-west-2)
- kitchen verify ubuntu(us-west-2)
- kitchen destroy ubuntu(us-west-2)
./execute_kitchen_terraform.sh
This script will begin by checking if bundler is installed, and then installing the necessary ruby gems.
Successfully installed bundler-2.1.4
Parsing documentation for bundler-2.1.4
Done installing documentation for bundler after 2 seconds
1 gem installed
Fetching gem metadata from https://rubygems.org/.........
Fetching gem metadata from https://rubygems.org/.
Resolving dependencies..
…
Using kitchen-terraform 5.2.0
Bundle complete! 1 Gemfile dependency, 185 gems now installed.
Use `bundle info [gemname]` to see where a bundled gem is installed.
Next the script will test if the public/private keypair exists in the test/assets directory, if not, it will create the key pair.
checking if test/assets directory exists
Generating public/private rsa key pair.
Your identification has been saved in test/assets/id_rsa.
Your public key has been saved in test/assets/id_rsa.pub.
The key fingerprint is:
SHA256:0oryWP5ff8kBwQPUSCrLGlVMFzU0rL7TQtJSi6iftyo Kitchen-Terraform AWS provider tutorial
The key's randomart image is:
+---[RSA 4096]----+
| ooo*X= |
| ..o. *o |
| o . . o |
| o + o . |
| . +.S= . . |
| +.o+ = . |
| . +.. +.o . o |
| *E ...+.. + |
| . o+=+o. o.. |
+----[SHA256]-----+
Next, the script will test the terraform project, using the python terraform-compliance module, and features located in test/features.
The script begins by testing if the terraform project has been initialized, and if not, initializing the project.
Initializing the backend...
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...
Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
After terraform initialization, the script will execute ‘terraform plan’ and output the plan in json format. It will then test the terraform output against the features in the test directory.
Refreshing Terraform state in-memory prior to plan...
The refreshed state will be used to calculate this plan, but will not be
persisted to local or remote state storage.
------------------------------------------------------------------------
An execution plan has been generated and is shown below.
Resource actions are indicated with the following symbols:
+ create
Terraform will perform the following actions:
# aws_instance.reachable_other_host will be created
+ resource "aws_instance" "reachable_other_host" {
+ ami = "ami-1ee65166"
+ arn = (known after apply)
+ associate_public_ip_address = true
+ availability_zone = (known after apply)
…
Plan: 11 to add, 0 to change, 0 to destroy.
------------------------------------------------------------------------
This plan was saved to: myout
To perform exactly these actions, run the following command to apply:
terraform apply "myout"
terraform-compliance v1.1.11 initiated
🚩 Features : /terraform-kitchen/test/features
🚩 Plan File : /terraform-kitchen/myout.json
🚩 Running tests. 🎉
Feature: security_group # /terraform-kitchen/test/features/security_group.feature
In order to ensure the security group is secure:
Scenario: Only selected ports should be publicly open
Given I have AWS Security Group defined
When it contains ingress
Then it must only have tcp protocol and port 22,443 for 0.0.0.0/0
1 features (1 passed)
1 scenarios (1 passed)
3 steps (3 passed)
You may be asking, why do we need both terraform-compliance features and kitchen-terraform fixtures for our testing? The purpose of terraform-compliance features is to have a repository of global, enterprise-level features and tests, which get applied to all projects. For example, the test displayed above will test security groups, so only ports 22 and 443 are open. No other ports should be open in the security group.
The kitchen-terraform fixtures and tests are designed for unit testing a single terraform project, and are not to be applied to every terraform project.
Continuing with the script execution, the script will now run the kitchen-terraform tests. It begins by attempting to destroy any existing terraform state in the applicable region.
-----> Starting Test Kitchen (v2.3.4)
-----> Destroying ...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.12.21
+ provider.aws v2.51.0
+ provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Initializing modules...
Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Selecting the kitchen-terraform-complex-suite-centos Terraform workspace...
$$$$$$ Finished selecting the kitchen-terraform-complex-suite-centos Terraform workspace.
$$$$$$ Destroying the Terraform-managed infrastructure...
module.complex_kitchen_terraform.random_string.key_name: Refreshing state... [id=none]
…
Destroy complete! Resources: 11 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-complex-suite-centos Terraform workspace...
Deleted workspace "kitchen-terraform-complex-suite-centos"!
$$$$$$ Finished deleting the kitchen-terraform-complex-suite-centos Terraform workspace.
Finished destroying (3m31.75s).
-----> Test Kitchen is finished. (3m32.88s)
The script will then initialize the terraform working directory and select a new terraform workspace.
-----> Starting Test Kitchen (v2.3.4)
-----> Creating ...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.12.21
+ provider.aws v2.51.0
+ provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Upgrading modules...
- complex_kitchen_terraform in ../../..
Initializing the backend...
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-complex-suite-centos Terraform workspace...
Created and switched to workspace "kitchen-terraform-complex-suite-centos"!
You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
$$$$$$ Finished creating the kitchen-terraform-complex-suite-centos Terraform workspace.
Finished creating (0m16.81s).
-----> Test Kitchen is finished. (0m17.97s)
The next step in the script is to run the ‘kitchen converge’. This step will converge the platforms in the kitchen.yml file.
-----> Starting Test Kitchen (v2.3.4)
-----> Creating ...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.12.21
+ provider.aws v2.51.0
+ provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Upgrading modules...
- complex_kitchen_terraform in ../../..
Initializing the backend...
Initializing provider plugins...
- Checking for available provider plugins...
- Downloading plugin for provider "random" (hashicorp/random) 2.1.2...
- Downloading plugin for provider "aws" (hashicorp/aws) 2.51.0...
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory.
$$$$$$ Creating the kitchen-terraform-complex-suite-centos Terraform workspace...
Created and switched to workspace "kitchen-terraform-complex-suite-centos"!
You're now on a new, empty workspace. Workspaces isolate their state,
so if you run "terraform plan" Terraform will not see any existing state
for this configuration.
$$$$$$ Finished creating the kitchen-terraform-complex-suite-centos Terraform workspace.
Finished creating (0m16.81s).
-----> Test Kitchen is finished. (0m17.97s)
Finally, the script will execute ‘kitchen verify’ to test the deployed project against the test suite.
-----> Starting Test Kitchen (v2.3.4)
-----> Setting up ...
Finished setting up (0m0.00s).
-----> Verifying ...
$$$$$$ Reading the Terraform input variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform input variables from the Kitchen instance state.
$$$$$$ Reading the Terraform output variables from the Kitchen instance state...
$$$$$$ Finished reading the Terraform output variables from the Kitchen instance state.
-----> Starting verification of the systems.
$$$$$$ Verifying the 'local' system...
Profile: complex kitchen-terraform (complex_suite)
Version: 0.1.0
Target: local://
✔ state_file: 0.12.21
✔ 0.12.21 is expected to match /\d+\.\d+\.\d+/
✔ inspec_attributes: static terraform output
✔ static terraform output is expected to eq "static terraform output"
✔ static terraform output is expected to eq "static terraform output"
Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 3 successful, 0 failures, 0 skipped
$$$$$$ Finished verifying the 'local' system.
…
$$$$$$ Finished verifying the 'remote' system.
$$$$$$ Verifying the 'remote2' system...
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_vpc' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:11)
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_subnets' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:16)
DEPRECATION: AWS resources shipped with core InSpec are being moved to a resource pack for faster iteration. Please update your profiles to depend on git@github.com:inspec/inspec-aws.git . Resource 'aws_security_group' (used at /private/tmp/terraform-kitchen/test/integration/complex_suite/controls/aws_resources.rb:22)
Profile: complex kitchen-terraform (complex_suite)
Version: 0.1.0
Target: aws://
✔ aws_resources: VPC vpc-00aa64d66abfa8e9c
✔ VPC vpc-00aa64d66abfa8e9c is expected to exist
✔ VPC vpc-00aa64d66abfa8e9c cidr_block is expected to eq "192.168.0.0/16"
✔ EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" states is expected not to include "pending"
✔ EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" cidr_blocks is expected to include "192.168.1.0/24"
✔ EC2 VPC Subnets with vpc_id == "vpc-00aa64d66abfa8e9c" subnet_ids is expected to include "subnet-000c991d9264c3a5f"
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to exist
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"198.144.101.2/32", :port=>22}
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"73.61.21.227/32", :port=>22}
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"198.144.101.2/32", :port=>443}
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f is expected to allow in {:ipv4_range=>"73.61.21.227/32", :port=>443}
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f group_id is expected to cmp == "sg-0bcdd1f63ba2a4b6f"
✔ EC2 Security Group sg-0bcdd1f63ba2a4b6f inbound_rules.count is expected to cmp == 3
✔ EC2 Instance i-0db748e47640739ea is expected to exist
✔ EC2 Instance i-0db748e47640739ea image_id is expected to eq "ami-ae7bfdb8"
✔ EC2 Instance i-0db748e47640739ea instance_type is expected to eq "t2.micro"
✔ EC2 Instance i-0db748e47640739ea vpc_id is expected to eq "vpc-00aa64d66abfa8e9c"
✔ EC2 Instance i-0db748e47640739ea tags is expected to include {:key => "Name", :value => "kitchen-terraform-reachable-other-host"}
Profile Summary: 1 successful control, 0 control failures, 0 controls skipped
Test Summary: 17 successful, 0 failures, 0 skipped
$$$$$$ Finished verifying the 'remote2' system.
-----> Finished verification of the systems.
Finished verifying (0m43.58s).
-----> Test Kitchen is finished. (0m44.76s)
The last step in the script is the ‘kitchen destroy’. This will destroy all AWS resources instantiated for the test.
-----> Starting Test Kitchen (v2.3.4)
-----> Destroying ...
$$$$$$ Verifying the Terraform client version is in the supported interval of >= 0.11.4, < 0.13.0...
$$$$$$ Reading the Terraform client version...
Terraform v0.12.21
+ provider.aws v2.51.0
+ provider.random v2.1.2
$$$$$$ Finished reading the Terraform client version.
$$$$$$ Finished verifying the Terraform client version.
$$$$$$ Initializing the Terraform working directory...
Initializing modules...
Initializing the backend...
Initializing provider plugins...
Terraform has been successfully initialized!
$$$$$$ Finished initializing the Terraform working directory
…
module.complex_kitchen_terraform.aws_vpc.complex_tutorial: Destroying... [id=vpc-00aa64d66abfa8e9c]
module.complex_kitchen_terraform.aws_vpc.complex_tutorial: Destruction complete after 1s
Destroy complete! Resources: 11 destroyed.
$$$$$$ Finished destroying the Terraform-managed infrastructure.
$$$$$$ Selecting the default Terraform workspace...
Switched to workspace "default".
$$$$$$ Finished selecting the default Terraform workspace.
$$$$$$ Deleting the kitchen-terraform-complex-suite-centos Terraform workspace...
Deleted workspace "kitchen-terraform-complex-suite-centos"!
$$$$$$ Finished deleting the kitchen-terraform-complex-suite-centos Terraform workspace.
Finished destroying (2m47.02s).
-----> Test Kitchen is finished. (2m48.17s)
Now the scripts will perform the same steps with ubuntu instances in us-west-2 region.
Future of Infrastructure Testing and Standards
In summary, I hope you have enjoyed this four-part series regarding infrastructure testing. While these articles only covered specific situations and scenarios for infrastructure testing and deployments, I hope it causes your organization to open a discussion about the future direction of infrastructure testing and standards.
Read the Entire DevOps Testing Series
Part I: Does DevOps Need Dedicated Testers?
Part II: 2019 Cloud Breaches Prove DevOps Needs Dedicated Testers
Part III: Practical Examples of DevOps Unit Testing
Part IV: More Complex Examples of DevOps Unit Testing
