Domain Driven Architecture

24. Juni 2018 By: Lukas Schondorff

Testing the cloud with dda-cloudspec

Testing entire infrastructures can be challenging especially when the infrastructure consists not only of container and server images.

Consequently, testing networks is the next important step in the process of testing infrastructures. We are currently using our dda-serverspec-crate for executing simultaneous tests on localhost or remote hosts by ssh on mulitple targets. Beside testing installed files or packages we've added the ability to do networking tests in addition. Imagine executing curl or netcat on a remote server connected over ssh.

If we combine the network testing ability with terraforms power of creating whole infrastructures, we will be able to spawn small probe servers in every corner. These probes will be our key-hole for inspecting our infrastructure. Ideally we want a tool that combines the power of terraform and allows for infrastructure testing. In order to achieve this, we need to handover the ip of the newly created probes as targets for the test execution. To do the integration work, we use mach. Mach is a remake of make, more modern, lightweight but easy to extend using clojure script. You can find our working example at dda-cloudspec.

This blog will guide you step by step through every tool and how to combine them. At the end you will be able to setup your own test in a similar way. For the impatient, the result could look like this:

Prepare your Environment

  1. You should have ssh client installed and a public ssh-key stored at ~/.ssh/
  2. Your aws tools should be installed & credentials configured
        apt install awscli
        vi ~/.aws/credentials
        aws_access_key_id = ACCESS_KEY
        aws_secret_access_key = SECRET_KEY
    Replace ACCESSKEY and SECRETKEY with your values. If you are using the more secure mfa setup, this should work fine.
  3. Install terraform
        curl -L -o /tmp/
        cd /tmp
        mv terraform /usr/local/bin/
  4. Install mach
        sudo apt-get install npm
        sudo npm install -g @juxt/mach
        sudo bash -c "cd /usr/local/bin && curl -fsSLo boot && chmod 755 boot"
  5. Test your mach installation: Create a file called Machfile.edn with the following content in it:
          testfunktion (println "Lets get started with dda-cloudspec!")
    and execude mach testfunktion in your bash you get the following output:
        Lets get started with dda-cloudspec!
  6. Clone dda-cloudspec
    git clone

Inspect Testdefinition

We've already prepared a dda-serverspec test configuration at main/ddapallet/serverspec.edn, also available in the dda-cloudspec which you cloned earlier:

 :file [{:path "/root/.bashrc"}
          {:path "/root/.profile" :mod "644" :user "root" :group "root"}]
 :netcat [{:host "" :port 80}
            {:host "" :port 80 :reachable? false}]

The ":file" keyword is testing for files located at your probe and ":netcat" tries to connect to the network. You can find a more complete overview of all available tests at dda-serverspec-crate.

Inspect the Integration of terraform and dda-serverspec

We have realized the integration of terraform & dda-serverspec in the Machfile.edn:

  {depends [terraform-plan]
   product (str terraform-build-dir "/output.json")
   update! #$ ["cd" terraform-build-dir "&&"
               "terraform" "apply" "-auto-approve" "-input=false" "proposed_apply.plan" "&&"
               "terraform" "output" "-json" ">" "output.json"]
   clean! #$ ["rm" product]}
If you allready familiar with terraform you will see, that we are using terraforms json output. We've to expose the ip as output - in main/terraform/ :
output "instance_public_ip" {
  value = "${aws_instance.instance.public_ip}"

Now we can easily parse terraforms output and fill the ip into our probe target definition Machfile.edn :

{depends [mk-pallet-build-dir]
 product (str dda-pallet-build-dir "/targets.edn")
 update! (dda.template-js/generate-with-terraform-output
              ( "templates/targets.edn.templ")
              (str terraform-build-dir "/output.json"))}

In the used template we couple the structure of to our targets definition, {{[instance_public_ip value]}} expresses the coupling path in terraforms nested output map - main/resources/templates/targets.edn.templ :

{:existing [{:node-name "test-vm1"
             :node-ip "{{[instance_public_ip value]}}"}]
 :provisioning-user {:login "ubuntu"}}

Spawn Probe and execute the Test

Now we're ready to start our test. Navigate to the folder, where you cloned dda-cloudspec and type:

mach test

The result should look like the following:

If a test fails, (eg. after you have removed .bashrc or .profile files in roots folder) your output may look like:

When we are done testing, you may want to destroy your probe to cut your expenses. So you can execute:

mach terraform-destroy

Start Testing your Infrastructure

So far, we showed the tip of the iceberg. We're sure there is much more to test. To learn more about the dda-cloudspec you can visit our github page. Your feedback and your ideas will be very welcome.

Tags: dda-cloudspec mach howto terraform dda-serverspec-crate dda-pallet

Powered by Cryogen | Free Website Template by Download Website Templates | Datenschutzerklärung | Impressum