To prove that DevOps needs a tester, you have to look no further than IdentityForce.com's biggest breaches of 2019 and review the types of breaches involved, and investigate why they occurred.
If you notice, a large percentage of the breaches were related to misconfiguration of cloud storage, and the lack of multi-factor authentication to access systems.
So who is primarily at fault: development, operations, security, networking, or DevOps?
While there could be many reasons for 'open' cloud storage and single-factor authentication to systems, I would suggest these are DevOps-related mistakes, and DevOps failed to: (1) properly test the security configuration of cloud storage prior to deployment, and (2) also failed to set up multi-factor authentication for accessing systems, and scan images for proper authentication to systems.
Some may argue that operations, security and/or networking departments are at fault, but the last line of defense before deployment is the Continuous Integration/Continuous Delivery (CI/CD) pipeline, which should include the application of common rule-sets and tests and is primarily the responsibility of DevOps.
Others will argue, proper CI/CD tools, such as Terraform, Sentinel, or Nexpose, or setting-up AWS config rules and using OpsWorks will prevent these issues; and they would be partially correct. These tools provide a layer of security and protection which is similar to application vulnerability scanning tools, but they do not replace unit-testing or integration testing.
The purpose of unit testing is to ensure the actual results match the expected results. Using public cloud storage as an example, the infrastructure project to create the cloud storage should contain unit-tests which include a:
Upon deployment of the project, the CI/CD pipeline will execute the unit-test, and if passing, perform integration testing.
The purpose of integration testing is to test individual units combined as a group. From an infrastructure perspective, this means the testing of cross-boundary access, permissions, and functionality. Utilize the public cloud storage example, and assuming the cloud storage had a permission to allow another account access to the storage, there would need to be an integration test for an external account to access the cloud storage – but who writes this code, and how do they know they need to write it?
This is where the concept of a DevOps tester is most applicable. Two separate infrastructure projects have been deployed; one for an account which has a dependency on cloud storage in a separate account, and one for cloud storage in a separate account. Ideally, DevOps should have recognized the dependency when creating the account, and created a unit-test that tests the permission of a mocked-up storage account. Someone would then need to write a separate integration test that is run in the CI/CD pipeline upon completion of both deployments.
Managing the inter/intra project dependencies, ordering, and priority of various infrastructure projects could become very overwhelming for DevOps, and is one of the primary reasons a DevOps tester is needed. Currently, I'm only seeing minimal infrastructure unit-testing, and not seeing any coordinated integration testing across infrastructure projects.
Just like when developers first began performing unit and integration testing, they performed these functions themselves. As the need arose, organizations would hire a software tester, and software testers would take more and more of the testing responsibilities; until software testing fully matured. DevOps is no different than normal software development and is still maturing as a concept.
As Infrastructure as Code becomes the norm, unit-testing and integration testing will become more common. Eventually, we will mature to a point where we are evaluating infrastructure code for code-quality, and preventing deployments that do not meet quality gateways.
The bottom-line: Infrastructure as Code will eventually mature to include unit-tests and integration testing, and become very similar to a normal software development lifecycle. Organizations should begin to further refine their own strategy on how this maturation will occur, and who will be responsible for the infrastructure testing.
In the next article, I provide Practical Examples of DevOps Unit Testing.
Part I: Does DevOps Need Dedicated Testers?
Part II: 2019 Cloud Breaches Prove DevOps Needs Dedicated Testers
