How to Implement Declarative Deletion

At meshcloud we've implemented a declarative API and the biggest challenge has been the declarative deletion of objects.

That's why in this blog post I want to answer the question:

How do I implement deletion for a declarative API?

Ahead I cover the challenges we ran into, how other systems solve it and which solution we applied at meshcloud.

Shoot through to my blog post about implementing a declarative API if you want to start at the beginning of this two part endeavour.

For the topic of declarative deletion let's start by having a look at the advantages of declarative deletion:

Why handle deletion declaratively?

If your use-case fits a declarative API you should also think about the deletion of objects.

If an external system syncs objects into your system and also ensures that objects are deleted when they are no longer present in the primary system, a declarative API simplifies client code a lot.

If deletion could only be executed in an imperative way, the client would have to find out which objects actually have to be deleted in the target system to call the delete endpoint for those objects accordingly.

Let's have a look at the group synchronization process:

As you can see the client somehow needs to build a diff between the desired state and the actual state in the target system to determine which objects have to be deleted. Moving this complex logic to the server side extracts this high implementation effort to only do it once instead of n-times for different clients.

Additionally it can provide a big performance improvement as quite a lot of data might be needed to do the diff. If that data only needs to be handled by the backend, you can get rid of network latency and bandwidth limitation for getting that data to the client.

In case of handling it at the client you may also struggle with outdated data, as getting the current state and processing it takes some time during which the state may already have changed in the target system.

How to implement deletion for a declarative API

The central conceptual question for deletion is

"How can you identify which objects need to be deleted?".

As not only one client will be using your API, a central aspect is to group objects into a Declarative Set. If one item in this set is missing, it will be deleted.

But items from another set are untouched. For understanding how to solve declarative deletion, let's have a look at actual implementations of it that are productively in use.

Another implementation challenge is an efficient algorithm for comparing the actual state with the desired state. This topic isn't covered in this blog post. Aspects like reducing the amount of DB queries, doing bulk queries for the objects that are part of the Declarative Set as well as bulk deletions are things that should be considered.

How do existing declarative systems handle declarative deletion?

Terraform

In Terraform, you define a tf file that contains the desired state. When this state is applied via terraform, it writes atfstate locally or to a shared remote location. That way Terraform knows which state is actually expected in the target systems. It can create a diff between the actual state (tfstate) and what is expected to be the new state (tf ). This requires an up-to-date state file. It deletes all resources that have been present in tfstate but are no longer present in tf.

The tf file is basically what I described as a Declarative Set before. So in case of a shared remote location that keeps track of the state, it is always related to the tf file and will ever only delete resources that are in the related tfstate.

Terraform also provides a terraform plan command, which will show you the actual changes Terraform would apply. That way you can verify whether the intended changes will be applied.

Additionally Terraform provides an imperative CLI command to delete specific resources or all resources in a tf file (-target can be defined optionally to delete specific resources).

terraform destroy -target RESOURCE_TYPE.NAME -target RESOURCE_TYPE2.NAME

Kubernetes

The recommended approach in Kubernetes is using the imperative kubectl delete command to delete individual objects. They recommend it, because it is explicit about which objects will be deleted (either a specific object or the reference to a yaml file).

But Kubernetes also supports the declarative approach. You have to kubectl apply a complete configuration folder. Additionally you have to use the alpha option --prune that actually allows deleting all objects that are no longer present in the yaml files inside the folder (see Kubernetes docs). The different ways of object management in Kubernetes are described very well here.

The kubectl apply command also provides an option for a dry-run to verify whether the intended changes will be applied.

How Kubernetes actually handles declarative deletion is explained very detailed in their documentation.

Here's a brief summary:

Kubernetes saves the complete state on server-side. When doing a kubectl apply --prune you have to either provide a label with -l <label> or refer to all resources in the namespaces used in the folder's yaml files via --all. Both ways match to what I described as a Declarative Set before. They make it possible for Kubernetes to know which objects actually belong to the set of resources that are part of the intended desired state. So when you apply again e.g. with the same label, it will simply delete all objects with this label, if they don't exist in the folder anymore. Using the label is the safer approach, as just deleting every no longer present resource in all namespaces that are referenced in a tf file is rather dangerous.

Also regarding what needs to be updated in an object, Kubernetes uses an interesting approach. It sets the actual configuration that was present in the yaml file into a last-applied-configuration metadata on the object. That way it will only e.g. delete attributes that have been present in a previous application but not anymore. It does not overwrite other attributes that have only been set via implicit commands. So it does an actual patch, based on what was before and is currently present in the yaml file.

The determination of what changes need to be applied has traditionally been done in kubectl CLI tool. But recently a server-side apply implementation has been released as GA.

Sadly they didn't touch multi-object apply yet. It remains in the kubectl CLI. So actual declarative object deletion is not part of the server-side implementation.

AWS Cloud Formation

AWS Cloud Formation uses so called Stacks in which they are grouping created resources. The Stack is what I called Declarative Set before. You can update an existing stack by applying a Cloud Formation Template to that Stack. You can modify the template and re-apply it. This modification can also contain removing resources. They will then be deleted in AWS when the Cloud Formation Template is applied to the Stack again.

How does meshStack handle it?

In meshStack we provide an API that takes a list of meshObjects and creates and updates them. If you want to apply declarative deletion you have to provide a meshObjectCollection. This is how we decided to implemenent the Declarative Set I mentioned before. This is similar to adding a label to kubectl apply or using a Stack in AWS. meshObjects no longer present in the request will be deleted if they had been applied before using the same meshObjectCollection parameter. The check in our backend is rather simple as we just added a meshObjectCollection field to our entities and can query by it. That way we can do a diff between what is applied in the request and what existed already before.

When actually executing the import of meshObjects, the result contains information about whether meshStack was able to successfully reconcile the desiredState or which error occurred for a specific meshObject. A failed import of one meshObject won't result in breaking the complete process. The response will only contain details about the import result of every single meshObject.

Currently we only support access to our API for a few external systems that integrate with meshStack. Once we provide a CLI and every user can use the API that way, meshObjectCollections will be assigned to projects. That way a clear separation of access to meshObjectCollections can be guaranteed.

Scalability of this approach is definitely a topic we will have to solve in future. With the current approach all objects have to be applied in a single request. If a huge number of objects shall be processed during one request, timeouts or other performance issues could arise. A possible solution we are thinking about is doing it similar to Kubernetes. You can define your intended objects in several yaml files inside a folder. Those can be uploaded to meshStack via a CLI or an according REST call. Processing of these objects will be done asynchronously once all files are uploaded.

When to use declarative deletion

Deletion support in a declarative API can be a really nice comfort feature for your clients, as a lot of complexity will be removed for them.

Removing no longer existing objects in the request will be handled by the declarative system.

The downside of declarative deletion is that it is implicit and can easily result in removing objects that were not intended to be removed, just because they were not part of the request anymore. As long as you are managing stateless objects it might be fine to take that risk as you could simply recreate them with the next request. If objects are stateful (e.g. a databases or volumes), it might be a bad idea to remove the resource and recreate it again. In that case all data will be gone. But even in the stateful use-cases you can reduce the risk by:

• making declarative deletion explicit via an additional parameter that needs to be provided, so the client is actually aware that declarative deletion will be applied.
• exclude certain objects from deletion (e.g. volumes and databases). Those can only be deleted in an imperative way. It could also be an option to let the end-user decide which objects should be excluded from the declarative deletion by flagging them accordingly.
• implement a dry-run functionality like terraform plan that will show the client which state will be applied in the end. This is a good option when providing access to the declarative API via a CLI for example. In case of an automated integration between two systems, it is not helpful as there is no-one to check the result of the dry-run. Still, some automation might be possible, but that would again require some complex logic on the client side, which we wanted to avoid in the first place.

In general it makes sense to additionally provide an imperative deletion, as the declarative deletion should always be an opt-in the client explicitly has to choose. The client always needs to be aware of the consequences the declarative deletion implies and that the client might need to be extra careful to always provide a complete list of resources.

by Stefan Tomm

Should I provide a declarative API? (You probably should)

Declarative APIs are becoming more and more popular, especially in the context of Infrastructure as Code.

At meshcloud we've implemented a declarative API. In this post I want to provide insights into the process and answer these questions:

• Does it make sense to provide declarative APIs for all systems?
• Which use-cases benefit from it and which don't?

But first things first, let's start with a look at what a declarative API actually is all about:

What is a declarative API?

At first let's have a look at the classical way of implementing an API. That is implementing it imperatively. With imperative APIs you have to provide dedicated commands the system has to execute: Create this VM, update the memory settings of this VM, remove a certain network from this VM, etc.

A declarative API is a desired state system. You provide a certain state you want the system to create. You don't care about all the steps needed to achieve that state. You just tell the system: "Please make sure that the state I provide will be there."

This approach is best known from Infrastructure as Code tools like Kubernetes or Terraform. You tell them that you want a certain set of resources with some given configuration. They take care of creating and updating the resources.

Why provide a declarative API?

With a declarative API you move complexity from the consumer of the system to the system itself. Creating, updating and even deleting objects is no longer a customer concern.

That means you can provide a way simpler API for your consumers by providing a declarative API to your system for some use-cases. This results in a reduced amount of errors due to misunderstandings between client and API provider.

It is for sure not the ideal solution to all use-cases, but more about that later.

Let's have a look at an example that shows how a declarative API can simplify the consumer's interaction with your system.

Example: Synchronizing Groups

Imagine you have a user group synchronized between a central Identity Provider (IdP) and your system (target). A group would have these properties:

Group:
    id = "123-456"
    displayName = "My Group"
    members = ["uid1", "uid2"]

Your system provides integrations for multiple IdPs and multiple clients of you API exist. These clients should focus on getting information from the IdP - and then on getting it into your target system.

Solving it with an Imperative API

Now imagine that you have an imperative API in the target system: What do you have to do to always keep all groups in sync?

Let's at first have a look at the operations that are available:

• createGroup: Creates a group with the given attributes. If a group already exists, an error is returned.
• updateGroup: Updates an existing group with the given attributes. If the group does not exist yet, it returns an error.
• deleteGroup: Deletes a group by its id.
• getAllGroups: Returns all groups that are available in the target system. This endpoint could provide some filter options, but this is not relevant for this blog post.

What you need to do for a full sync of groups from the IdP to your system:

As you can see, creating such a synchronization process is a rather complex task. Especially in the given example. You want a lightweight solution to implement multiple clients for the different IdPs.

This complexity requires a lot of effort every time you integrate a new IdP at a customer.

Solving it with a declarative API

In cases like the group sync, moving all the complex update logic to the target system and providing a declarative API simplifies the client by a lot.

How would a declarative API look like?

• applyGroups You only need one operation to which you provide a list of groups, like this:

[
    {
        id: "123-456"
        groupName: "developers",
        members: ["dev1", "dev2"]
    },
  {
        id: "456-789"
        groupName: "managers",
        members: ["manager1"]
    },
    {
        id: "789-132"
        groupName: "operators",
        members: ["op1"]
    }
]

On the next synchronization dev2 became a manager and has been moved to the managers group. In addition, the operators group has been removed, as the company decided to go with a DevOps team. So op1 has been moved to the developers group and the developers group has been renamed to devops. All you have to do is run the exact same process as before, which is:

That means the second call will be looking like this:

[
    {
        id: "123-456"
        groupName: "devops",
        members: ["dev1", "ops1"]
    },
  {
        id: "456-789"
        groupName: "managers",
        members: ["manager1", "dev2"]
    }
]

The target system will take care of removing dev2 from the developers group and will add it to the managers group. It will rename the developers group to devops and add member ops1 to that group. It will also take care of removing the operators group.

For sure, all the logic mentioned in the imperative approach now needs to be implemented in your target system. BUT you will only have to implement it once.

You may even come up with an architecture of your system that simplifies implementation of that declarative approach further.

This example is limited to a holistic synchronization of all groups all the time. If you have multiple sources you get your input from, you need some kind of bucket for the groups coming from one system. You could e.g. simply add another input to the applyGroups function that allows submitting an additional bucket parameter. That allows your system to only take all groups related to the given bucket into consideration for the consolidation of which groups to create, update or delete.

As this is especially relevant for deleting groups, more details about this will be part of my upcoming blog post on "How to implement deletion for a declarative API?".

The as-code advantage

Another great advantage of the declarative approach is that you can store what you applied in a version control system (VCS). That approach provides you several advantages.

• You have a full history of all changes
• You have a nice overview of the expected state in a system.
• You can work cooperatively on the desired state with a whole team.

Kubernetes or Terraform are good, real-world examples of storing the desired state in a VCS. The Kubernetes YAML files and the Terraform files are usually stored in a VCS.

If in contrast, you only apply imperative commands to the target system, you would always have to ask the target system about the current state. You may also override changes or reapply changes someone else had done before.

Declarative vs Imperative API, opponents or a nice team?

You may ask yourself: Should I build a completely declarative system without providing any imperative commands?

In some rare cases that might be the right thing to do. In most cases it makes much more sense to combine an imperative and a declarative API. Actually that is what the big Infrastructure as Code tools do. In the Kubernetes documentation you can find an imperative as well as a declarative way of managing your Kubernetes objects.

Besides use-cases like the group synchronization or Infrastructure as Code that profit a lot from a declarative system, there are also use-cases that benefit from an imperative API.

So you should decide dependent on the different use-cases in your system which API fits better for which case.

Example for less effort with Imperative API

Let's extend our previous example with the user-groups and say you need these groups to assign them to projects in your system. These projects cannot simply be created in your system, but a central workflow tool with an approval process must be used to create new projects. After the workflow completed, this tool will create the according project in your system via your API. Afterwards the project will only be maintained in your system. There will be no updates coming from the workflow tool.

An imperative approach with an operation createProject is just the right thing for that use-case. Sure, it would also work with a declarative approach. You'd just call the apply operation once to create the project. But as only a create operation will be done once, there is no need for the complex handling of updating or deleting existing projects. So you can save quite a lot of effort by not implementing the declarative handling in your system, if you won't use it.

Example for simpler client implementation with Imperative API

Another example is only updating a certain attribute of e.g. the project. Let's say the project has some tags that can be set on it. Tags are simple key/value pairs. If you want to update them in a declarative API, you have to provide the complete project like this:

Project:
    id: "123-456",
    displayName: "My Project",
    assignedGroups: [
        "456-789"
    ],
    tags: [
        environment: "dev"
    ]

But if a system - that only knows about the project id and the tags - provides the tags, how does it get the other information it needs to update the project? It would have to call a get endpoint on the target system first to get all data of the project first. Then it could set the new tags and update the project.

For this use-case an imperative API makes the client's life much easier. It could just call an imperative operation like addTag or setTags.

target addTag "environment" to "prod"
target setTags ["environment" to "prod", "confidentiality" to "internal"]

When should you provide a declarative API?

In the end you have to decide per use-case which kind of API design fits best. The following comparison provides some advantages of the 2 approaches. They can help you in making that decision.

Imperative API

• finer control on client side
• fits perfectly fine when only creating new objects and not keeping these objects in sync after creation
• easier to use when only updating partial data (especially when attributes of an object are sourced from multiple different systems)

Declarative API

• way easier client code for data synchronization scenarios as the client only needs to provide the desired state and the rest is done by the target system
• version control option for desired state
• single point of implementation at server side
• hide complexity of creating a certain state in the backend. Clients don't have to care about that complexity

In general, I think the following rule of thumb can also help with that decision:

• Use a declarative API if you want to keep objects in sync with a central source. This is especially true if you have multiple clients of that API, as the complex handling of updates and deletion only have to be implemented once centrally in your system.
• Use an imperative API for one-time operations. You just want to create, update or delete one specific thing and afterwards you don't care about the object's lifecycle anymore. In that case you should consider an imperative API.

But now I'd like to hear from you: Have you implemented a declarative API? What challenges did you come across? Did I miss anything in my blog post? Let me know!

I recommend you to visit our blog where you will find many interesting posts on engineering topics: e.g. our guides to TLS/SSL certificates or the guide to testing IaC.

by Stefan Tomm

A Developers Practical Guide to TLS/SSL Certificates

For many developers certificates are a black box: It is old tech with terrible documentation and the underlying encryption is complex and hard to understand. This practical guide to TLS/SSL certificates will help you navigate these hazardous waters.

In this guide you will learn:

• Reasons why a certificate may be invalid
• All about self signed certificates
• Certificate formats
• Components of a certificate

Why you should care about TLS/SSL certificates

Certificates play a central role in IT and cloud security. Failure to understand them and handle them correctly can result in serious damage to business:

• Certificates can expire causing outages - it happened to Microsoft
• Private keys could be exposed if handled incorrectly - Atlassian made that mistake
• Creating self-signed certificates that are unsafe
• Man-in-the-middle attack to retrieve passwords - not easily detectable
• Bad actors authenicating themselves with certificates - the NSA posed as Google

Certificate vs. public key vs. private key vs. CSR

When using TLS certificates you will be working with different parts that work together in different use cases:

Key pairs consist of private and public keys that belong together and are used for asymmetric encryption:

• Private key
  • must be kept secret (not part of a certificate)
  • decrypt messages encrypted with the respective public key
  • sign messages
• Public key
  • shared with others (included with a certificate)
  • encrypt messages for the holder of the respective private key
  • verify messages were signed by the respective private key

Certificate signing requests (CSR) are basically unsigned certificates. They contain all information required for creating a certificate including the public key (but not the private key). They're presented to a certificate authority (e.g. a well-known certificate issuer) which can then validate that everything is in order (e.g. if someone requests a certificate for example.com the certificate issuer should ensure that the requestor actually controls that domain), and use its own private key to create a signed certificate from the information contained in the CSR.

Think CSR = certificate - issuer signature.

A certificate contains many pieces of information with the following being of most practical importance:

• Subject: Who is the owner of the certificate?
• Subject Public key: Used for communicating with the certificate owner.
• Issuer: Who signed this certificate?

Certificates are usually not secret since they contain no private information, however, they're sometimes bundled together with their private keys, so care must be taken.

TLS certificates contain different pieces of information, you can look at the contents of a certificate using openssl. When working with certificates it's very likely that you will encounter the following parts:

Information about the certificate subject, who does this certificate belong to?

The Common Name (or CN) is the most basic piece of identifying information about the subject of a certificate and you may well encounter certificates that provide only CNs and no further subject information. Previously the common name was used to verify that a certificate was used for the correct host, so in most cases, certificates still use a hostname for the common name (as seen above).

CN validation may still work with legacy applications but current browsers and libraries are no longer using the common name for validation, instead, they use the Subject Alternative Name (or SAN).

Using SAN has the added benefit of working with a list of domain names and that you may also include IP addresses.

Issuer information about the certificate issuer, which authority signed this certificate?

$ openssl x509 -in example.com.pem -noout -text | grep 'Issuer:'
Issuer: C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1

It also contains the Authority Key Identifier which allows you to identify the key pair used for signing this certificate.

$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Authority Key Identifier'
X509v3 Authority Key Identifier:
keyid:B7:6B:A2:EA:A8:AA:84:8C:79:EA:B4:DA:0F:98:B2:C5:95:76:B9:F4

Formats of certificates

Common formats for TLS certificates are

• DER encoded binary data (.der, .cer, .crt)
• PEM: Base64 encoded DER (.pem but also often .cer and .crt), files begin and end with -----BEGIN CERTIFICATE-----/-----END CERTIFICATE-----. Used by most *nix applications/libraries.
• PKCS12: Binary data, may include private key, optionally password protected (.pfx, .p12).

Reasons why a certificate may be invalid

As a developer, you're most likely to get into certificates when something doesn't work. Usually, this is because a certificate can't be validated. Here are some of the most likely reasons why a certificate is invalid.

Expired or not yet valid

Certificates are only valid for a specific time window, so not only can they expire they may also be not valid yet.

# Lookup certificate validity for a local certificate.
$ openssl x509 -in example.com.pem -noout -text | grep -A2 'Validity'
Validity
  Not Before: Nov 24 00:00:00 2020 GMT
  Not After : Dec 25 23:59:59 2021 GMT

Some clients may also reject certificates that are valid for too long periods (typically 1 or 2 years), e.g. Safari does not trust certificates that are valid for more than 1 year plus a grace period of 30 days.

Let's Encrypt certificates are also only valid for 90 days which is why they provide tools for automated renewals.

Name validation fails

TLS certificates are only valid for a specific set of domains/addresses which are specified in a certificate's Subject Alternate Name (SAN) field.

# Looking up SAN of a local certificate
$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Subject Alternative Name:'
X509v3 Subject Alternative Name:
  DNS:www.example.org, DNS:example.com, DNS:example.edu, DNS:example.net, DNS:example.org, DNS:www.example.com, DNS:www.example.edu, DNS:www.example.net

The field contains a list of DNS names and/or IP addresses that this certificate is valid for. DNS names may also contain wildcards like *.example.com but note that this does not cover additional subdomains like foo.bar.example.com or the naked domain name example.com.

Since SAN was not always present in certificates you may encounter legacy applications that still rely on validating against the Common Name (CN) of a certificate, which is why this is usually set to the domain name as well.

# Looking at the subject information of a local certificate
$ openssl x509 -in example.com.pem -noout -text | grep 'Subject:'
Subject: C = US, ST = California, L = Los Angeles, O = Internet Corporation for Assigned Names and Numbers, CN = www.example.org

When working with recent applications and libraries setting the common name is not enough and you definitely need to specify SAN.

No trust

Certificates are not trusted automatically but rely on a big bundle of trusted certificate authorities (CA) that are usually distributed as part of an operating system or firmware. Your operating system or device is configured to trust all certificates signed by these CAs. If you encounter a certificate that is signed by a different CA it will be rejected.

Applications may also decide to use their own sets of trusted certificates, e.g. the Java Keystore.

If a certificate is not trusted even though it has been issued by a well-trusted issuer there may be missing intermediate certificates.

Consider the following example: a trusted CA A has issued a certificate to another CA B which allows them to issue certificates of their own. The certificate you're seeing has been issued by B and since B is not included in your trusted certificates the certificate is rejected. To avoid this issue everyone using certificates issued by B should also include the certificate which verifies that B may issue certificates (issued by A). This is an intermediate certificate that allows clients to verify that there is an intact chain of trust from A to B to the certificate they're actually concerned with.

It's of course also possible to encounter certificates from CAs that are simply not trusted e.g. because they're only used for private or internal purposes. If you are certain that such a CA should be trusted you can of course add them to your locally trusted CAs.

Forbidden usage

Certificates can be restricted in their usage, e.g. when you receive a signed certificate from a well-known issuer you're not allowed to sign other certificates, i.e. you may not act as a CA yourself because then you could sign any certificates you wanted to without oversight.

# Checking allowed usage
$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Usage'
X509v3 Key Usage: critical
  Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
  TLS Web Server Authentication, TLS Web Client Authentication

Above you see the typical set of allowed usages for a TLS certificate used for HTTPS, note that certificate signing is not included.

Self-signed certificates

Self-signed certificates are frequently used for testing purposes or in ad-hoc situations. They're signed by the same key that is used for the certificate itself. This implies that the certificate metadata has not been verified by an external entity which is why they're considered not trustworthy.

To learn more about the meshcloud platform, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to getting in touch with you.

by Henry Dettmer

Testing Infrastructure as Code (IaC): The Definitive Guide 2020

In this blog post we're going to explain if and how Infrastructure as Code should be tested. We'll illustrate 5 examples with Terraform - the tool we use here at meshcloud - and tell you what to look for in IaC test tooling.

Here are the topics we will touch on. Let's dive right in:

1. What is Infrastructure as Code?

2. Do I Even Need to Test IaC?

3. The IaC Testing Usefulness Formula

4. The Developer Effect

5. The 3 Essential Types of Testing Infrastructure as Code

6. 4 Practical Examples for IaC Testing

7. The 3 Key Factors for Choosing Your Test Tooling

What Exactly is Infrastructure as Code?

To make sure we are starting from the same point: What do we mean by Infrastructure as Code (IaC)? IaC describes the specification of required infrastructure (VMs, storage, networking) in text form. We define a target state, which can be easily adapted, duplicated, deleted and versioned. IaC relies on modern cloud technologies and enables a high degree of automation. With these new trends, infrastructure development has become much more similar to application development. This raises the interesting question if and how you can test infrastructure code?

Do I Need to Test IaC at All?

The question is more relevant than you might think. Modern IaC tools like terraform or pulumi already have a lot of checks built-in and respond with detailed error messages. In order to better asses the usefulness of IaC testing for you, consider these factors:

1. Number of components (e.g. vms, managed services, loadbalancers, etc.)
2. Number of environments (e.g. dev, stg, prod)
3. Number of Rollouts which affect Infrastructure (daily releases vs. monthly)

The IaC Testing Usefulness Formula

We believe all three factors - components, enviroments and number of rollouts - are equally important, which results our IaC Testing Usefulness Formula:

Usefulness of IaC Testing = Components x Environments x Rollouts

The Developer Effect

In practice we observed another effect which significantly increases the number of rollouts. Lovingly dubbed the "devEffect" at meshcloud it occurs during the initial set-up of IaC or the modification of existing components. Because developing new IaC is often done in a trial and error manner, developers will rollout their changes around 100 times more often. For this reason alone it can be worth it to write IaC tests.

The 3 Essential Types of Testing Infrastructure as Code

1. Static and local checks
2. Apply and destroy tests
3. Imitations tests

1. Static and Local Checks

Strictly speaking these are not tests, but simple checks. However we still want to include them in this list as they are an easy and fast way to ensure the correct set-up of your infrastructure.

Idea: Analyze configurations and dependencies before execution.

Goal: Fast detection of static errors. Errors that occur dynamically during execution are not covered.

Examples: Completeness check, field validation, reference analysis.

But everything is better with code examples, so let's have a look. Consider the following terraform file to create a small VM.

resource google_compute_instance test {
  name         = "hello-terratest"
  machine_type = "f1-micro"

  boot_disk {
    initialize_params {
      image = "ubuntu-1804-lts"
    }
  }

  network_interface {
    network = "default"
    access_config {}
  }
}

The alert reader already noticed the absence of the machine_type field. You didn't? That is exactly our point: You don't have to. Terraform automatically informs us about the missing field.

This can be taken even a step further in the form of autocomplete. Using the terraform language server and VSCode for example, it automatically prompts me to enter the missing field and its type.

Additionally we can also check for static dependencies. Consider the following terraform code. We assume this is the only code in the module. It references a google compute target pool which does not exist within the context. This is a total lifesaver in projects with hundreds of dependencies.

resource google_compute_forwarding_rule test {
  name   = "hello-terratest"
  target = google_compute_target_pool.test.self_link
}

While static checks are already a big step-up from traditional infrastructure deployments, they do not account for no dynamic fields and dependencies. This is where our next category Apply and Destroy Tests comes into play

2. Apply and Destroy Tests

With Apply and Destroy Tests we can go one step further and also identify dynamic errors.

Idea: Roll out the infrastructure for a short time, test it and destroy it again immediately afterwards.

Goal: Check dynamic fields and dependencies.

Examples: IP addresses, IDs, random generated passwords.

For these type of tests we are currently working with terratest. Terratest is a collection of go libraries that simplify the interaction with IaC providers and well-known cloud providers. For the following example we are using the terraform module. We can see that terratest easily integrates into terraform and is able to send commands and extract output from them.

package test

import (
    "testing"

    "github.com/gruntwork-io/terratest/modules/terraform"
)

func TestApply(t *testing.T) {
    t.Parallel()

    tfOptions := &terraform.Options{
        TerraformDir: "./terraform/apply_test",
    }

    defer terraform.Destroy(t, tfOptions)

    terraform.InitAndApply(t, tfOptions)

    // further validations go here

}

Up- and downsides to consider:

Read more

by Wulf Schiemann