Testing entire infrastructures can be challenging especially when the infrastructure consists not only of container and server images.

Consequently, testing networks is the next important step in the process of testing infrastructures. We are currently using our dda-serverspec-crate for executing simultaneous tests on localhost or remote hosts by ssh on mulitple targets. Beside testing installed files or packages we've added the ability to do networking tests in addition. Imagine executing curl or netcat on a remote server connected over ssh.

If we combine the network testing ability with terraforms power of creating whole infrastructures, we will be able to spawn small probe servers in every corner. These probes will be our key-hole for inspecting our infrastructure. Ideally we want a tool that combines the power of terraform and allows for infrastructure testing. In order to achieve this, we need to handover the ip of the newly created probes as targets for the test execution. To do the integration work, we use mach. Mach is a remake of make, more modern, lightweight but easy to extend using clojure script. You can find our working example at dda-cloudspec.

This blog will guide you step by step through every tool and how to combine them. At the end you will be able to setup your own test in a similar way. For the impatient, the result could look like this:

Prepare your Environment

1. You should have ssh client installed and a public ssh-key stored at ~/.ssh/id_rsa.pub
2. Your aws tools should be installed & credentials configured

       apt install awscli
       vi ~/.aws/credentials
       [default]
       aws_access_key_id = ACCESS_KEY
       aws_secret_access_key = SECRET_KEY
       

   Replace ACCESSKEY and SECRETKEY with your values. If you are using the more secure mfa setup, this should work fine.
3. Install terraform

       curl -L -o /tmp/terraform_0.11.7_linux_amd64.zip https://releases.hashicorp.com/terraform/0.11.7/terraform_0.11.7_linux_amd64.zip
       cd /tmp
       unzip terraform_0.11.7_linux_amd64.zip
       mv terraform /usr/local/bin/
       

4. Install mach

       sudo apt-get install npm
       sudo npm install -g @juxt/mach
       sudo bash -c "cd /usr/local/bin && curl -fsSLo boot https://github.com/boot-clj/boot-bin/releases/download/latest/boot.sh && chmod 755 boot"
       

5. Test your mach installation: Create a file called Machfile.edn with the following content in it:

       {
         testfunktion (println "Lets get started with dda-cloudspec!")
       }
       

   and execude mach testfunktion in your bash you get the following output:

       Lets get started with dda-cloudspec!
       

6. Clone dda-cloudspec

   git clone https://github.com/DomainDrivenArchitecture/dda-cloudspec.git
   

Inspect Testdefinition

We've already prepared a dda-serverspec test configuration at main/ddapallet/serverspec.edn, also available in the dda-cloudspec which you cloned earlier:

{
 :file [{:path "/root/.bashrc"}
          {:path "/root/.profile" :mod "644" :user "root" :group "root"}]
 :netcat [{:host "www.google.com" :port 80}
            {:host "www.google.c" :port 80 :reachable? false}]
}

The ":file" keyword is testing for files located at your probe and ":netcat" tries to connect to the network. You can find a more complete overview of all available tests at dda-serverspec-crate.

Inspect the Integration of terraform and dda-serverspec

We have realized the integration of terraform & dda-serverspec in the Machfile.edn:

terraform-apply
  {depends [terraform-plan]
   product (str terraform-build-dir "/output.json")
   update! #$ ["cd" terraform-build-dir "&&"
               "terraform" "apply" "-auto-approve" "-input=false" "proposed_apply.plan" "&&"
               "terraform" "output" "-json" ">" "output.json"]
   clean! #$ ["rm" product]}

If you allready familiar with terraform you will see, that we are using terraforms json output. We've to expose the ip as output - in main/terraform/output.tf :

output "instance_public_ip" {
  value = "${aws_instance.instance.public_ip}"
}

Now we can easily parse terraforms output and fill the ip into our probe target definition Machfile.edn :

{depends [mk-pallet-build-dir]
 product (str dda-pallet-build-dir "/targets.edn")
 update! (dda.template-js/generate-with-terraform-output
              (lumo.io/resource "templates/targets.edn.templ")
              (str terraform-build-dir "/output.json"))}

In the used template we couple the structure of output.tf to our targets definition, {{[instance_public_ip value]}} expresses the coupling path in terraforms nested output map - main/resources/templates/targets.edn.templ :

{:existing [{:node-name "test-vm1"
             :node-ip "{{[instance_public_ip value]}}"}]
 :provisioning-user {:login "ubuntu"}}

Spawn Probe and execute the Test

Now we're ready to start our test. Navigate to the folder, where you cloned dda-cloudspec and type:

mach test

The result should look like the following:

If a test fails, (eg. after you have removed .bashrc or .profile files in roots folder) your output may look like:

When we are done testing, you may want to destroy your probe to cut your expenses. So you can execute:

mach terraform-destroy

Start Testing your Infrastructure

So far, we showed the tip of the iceberg. We're sure there is much more to test. To learn more about the dda-cloudspec you can visit our github page. Your feedback and your ideas will be very welcome.