Dhall: Degrees of Correctness or Temperature?

Degrees of Correctness With Dhall

At meshcloud we are huge fans of infrastructure-as-code, GitOps and declarative approaches in general. This means we manage a lot of declarative files in our Git repository and these files get processed automatically when they change. Ideally we want our CI system to make sure that changes we’re making to these files won’t cause any problems once they reach a staging or production environment.

Some tools like Terraform allow you to run validation on your declarative definitions but it’s very likely that you’re also dealing with some JSON or YAML files (e.g. CI pipeline definitions, application configurations, Kubernetes manifests, etc.) that are required by an application that does not provide any way to check these files beforehand.

Let’s have a look how we can use Dhall to solve this problem and even improve on the limitations of formats like JSON or YAML.

Syntactic Correctness in Dhall

As a first step we need to make sure that any files we check in are syntactically correct, this can also be achieved by using something like a JSON/YAML linter but if we have everything as Dhall files we can get by with one tool and get some added benefits like being able to share code via imports, define constants, and use functions.

Let’s see an example, here is a YAML file containing configuration for an application:

roles:
  - name: admins
    permissions: admin
    priority: 100
  - name: developers
    permissions: editor
    priority: 10

Okay, there is a list of roles and each group has a name, permissions and a priority value. Now let’s make a change, we’ll add another role.

roles:
  - name: admins
    permissions: admin
    priority: 100
  - name: developers
    permissions: editor
    priority: 10
  - name: guests
   permissions: viewer
    priority: 1

Can you spot the mistake? By removing a single space before permissions we’ve now created an invalid YAML file so our application won’t be able to parse it and will crash!

Let’s write the same config in Dhall instead:

{ roles =
  [ { name = "admins", permissions = "admin", priority = 100 }
  , { name = "developers", permissions = "editor", priority = 10 }
  , { name = "guests", permissions = "viewer", priority = 1 }
  ]
}

Running this file through Dhall (dhall --file config.dhall) will let us know about any syntactic errors and if everything is correct we can be sure that we can use dhall-to-yaml to generate a valid YAML file for our application later (e.g. as part of our CD process).

Type Correctness in Dhall

So far so good, but what about other kinds of mistakes, for example:

{ rolos =
  [ { name = "admins", permissions = "admin", priority = 100 }
  , { name = "developers", permissions = "editor", priority = 10 }
  , { name = "guests", permissions = "viewer", priority = 1 }
  ]
}

Looks like we misspelled roles but Dhall does not complain since everything is syntactically correct!

To ensure that this does not happen we need to tell Dhall about the shape of our configuration. We do this by specifying a type in other languages this may be called a schema or a spec. It looks like this:

{ roles : List { name : Text
               , permissions : Text
               , priority : Natural
               }
}

Alright, instead of assigning values with = we’re now specifying types with :. Roles is a List with an inner type for the list members and this inner type has two text fields name and permissions and a field priority which should contain a natural number.

We can clean this up a bit by pulling the definition for the inner type into a let binding (basically a constant). By convention, types are given upper case names so let’s call the inner type Role:

let Role = { name : Text
           , permissions : Text
           , priority : Natural
           }

in  { roles : List Role }

This is the same definition as before just a bit more readable.

Now that we have our type written out we can combine it with our actual value. There are different ways to go about this but we’ll keep it simple and add a type annotation to our values directly:

let Role = { name : Text
           , permissions : Text
           , priority : Natural
           }

let Config = { roles : List Role }

in    { roles =
        [ { name = "admins", permissions = "admin", priority = 100 }
        , { name = "developers", permissions = "editor", priority = 10 }
        , { name = "guests", permissions = "viewer", priority = 1 }
        ]
      }
    : Config

First we move our type definition into its’ own let binding and call it Config. Then we add the same config values as before but we make sure to add an annotation of : Config which tells Dhall about the type this value should conform to.

Running it through dhall-to-yaml will yield the same result as before:

$ dhall-to-yaml --file config.dhall
roles:
  - name: admins
    permissions: admin
    priority: 100
  - name: developers
    permissions: editor
    priority: 10
  - name: guests
    permissions: viewer
    priority: 1

Introducing the same mistake as before will result in an error though:

let Role = { name : Text
           , permissions : Text
           , priority : Natural
           }

let Config = { roles : List Role }

in    { rolos =
        [ { name = "admins", permissions = "admin", priority = 100 }
        , { name = "developers", permissions = "editor", priority = 10 }
        , { name = "guests", permissions = "viewer", priority = 1 }
        ]
      }
    : Config
$ dhall --file config.dhall
Use "dhall --explain" for detailed errors

Error: Expression doesn't match annotation

{ - roles : …
, + rolos : …
}

 8│       { rolos =
 9│         [ { name = "admins", permissions = "admin", priority = 100 }
10│         , { name = "developers", permissions = "editor", priority = 10 }
11│         , { name = "guests", permissions = "viewer", priority = 1 }
12│         ]
13│       }
14│     : Config

config.dhall:8:7

As we expected the expression (our value) doesn’t match the annotation (our type), - roles tells us that the roles field is missing (hence the minus sign) and + rolos tells us that there was an unexpected additional field called rolos.

Great, we’ve achieved type correctness, our configuration will definitely be valid and every field will have the correct type!

Semantic Correctness

We’re not done yet though, what happens when we do something like this:

let Role = { name : Text
           , permissions : Text
           , priority : Natural
           }

let Config = { roles : List Role }

in    { roles =
        [ { name = "admins", permissions = "admin", priority = 100 }
        , { name = "developers", permissions = "editor", priority = 10 }
        , { name = "guests", permissions = "read-only", priority = 1 }
        ]
      }
    : Config

We’ve changed the guest role permissions to "read-only", it’s still a text value so our types are in order but when we feed this to our application we get an error:

$ my-app config.yml
ERROR: unknown permissions "read-only" expected one of "admin", "editor", "viewer"

Oh no, permissions isn’t actually a text value it’s an enum! There is no way to encode this in a YAML file so our application reads the text and tries to map it to an enum value of the same name. Even though all our types are in order we’ve failed to provide correct data because the underlying format (YAML) is not expressive enough to specify values that are semantically correct.

Luckily it’s very easy to cover this exact case with Dhall thanks to union types. Union types are used if you have a value that can be of different types and the simplest way of using it is to specify a type that consists of different specific text values.

We create a new Permissions union type as a let binding and use it in our definition of Role:

let Permissions = < admin | editor | viewer >

let Role = { name : Text
           , permissions : Permissions
           , priority : Natural
           }

Between the angle brackets < ... > we have the different values we want to allow (called constructors) separated by pipes |.

Since we’re no longer working with text values we also need to update our values. Instead of using text values like "admin" we write Permissions.admin. Here’s the full example:

let Permissions = < admin | editor | viewer >

let Role = { name : Text
           , permissions : Permissions
           , priority : Natural
           }

let Config = { roles : List Role }

in    { roles =
        [ { name = "admins", permissions = Permissions.admin, priority = 100 }
        , { name = "developers", permissions = Permissions.editor, priority = 10 }
        , { name = "guests", permissions = Permissions.viewer, priority = 1 }
        ]
      }
    : Config

By writing it in the form Type.constructor we can differentiate between alternatives of the same name, for example we could also have a Role.admin.

If we use a non-existent role like Permissions.read-only we’re greeted with an appropriate error:

$ dhall --file config.dhall
Use "dhall --explain" for detailed errors

Error: Missing constructor: read-only

11│                                            Permissions.read-only

config.dhall:11:44

Dhall error messages can be a bit hard to read but in this case the --explain flag does a pretty nice job:

$ dhall --explain --file config.dhall                                                                                                                                                                                                                          ~

Permissions : Type
Role : Type
Config : Type

Error: Missing constructor: read-only

Explanation: You can access constructors from unions, like this:

    ┌───────────────────┐
    │ < Foo | Bar >.Foo │  This is valid ...
    └───────────────────┘

... but you can only access constructors if they match an union alternative of
the same name.

For example, the following expression is not valid:

    ┌───────────────────┐
    │ < Foo | Bar >.Baz │
    └───────────────────┘
                    ⇧
                    Invalid: the union has no ❰Baz❱ alternative

You tried to access a constructor named:

↳ read-only

... but the constructor is missing because the union only defines the following
alternatives:

↳ < admin | editor | viewer >

────────────────────────────────────────────────────────────────────────────────

11│                                            Permissions.read-only

config.dhall:11:44

Now we’ve done pretty much all we can do to ensure that our configuration is as correct as possible and by using Dhall our CI system can inform us about such problems before they turn into failing deployments or runtime errors. Of course we can’t catch all possible errors before we actually run an application (e.g. we specify a wrong but still valid email address) but we should try to catch problems as early as possible.

Where to Go From Here

We’ve only scratched the surface!

  • Union types are much more expressive and each alternative can contain a full other type not just a fixed value.
  • Dhall can automatically embed type information in generated JSON/YAML.
  • Using packages and default values.
  • You can bypass more limitations of JSON/YAML by using Dhall to write you configuration in a way that makes sense semantically and then use a function to generate one or multiple output files.

A Developers Practical Guide to TLS/SSL Certificates

For many developers certificates are a black box: It is old tech with terrible documentation and the underlying encryption is complex and hard to understand. This practical guide to TLS/SSL certificates will help you navigate these hazardous waters.

In this guide you will learn:

  • Reasons why a certificate may be invalid
  • All about self signed certificates
  • Certificate formats
  • Components of a certificate

Why you should care about TLS/SSL certificates

Certificates play a central role in IT and cloud security. Failure to understand them and handle them correctly can result in serious damage to business:

Certificate vs. public key vs. private key vs. CSR

When using TLS certificates you will be working with different parts that work together in different use cases:

Key pairs consist of private and public keys that belong together and are used for asymmetric encryption:

  • Private key
    • must be kept secret (not part of a certificate)
    • decrypt messages encrypted with the respective public key
    • sign messages
  • Public key
    • shared with others (included with a certificate)
    • encrypt messages for the holder of the respective private key
    • verify messages were signed by the respective private key

Certificate signing requests (CSR) are basically unsigned certificates. They contain all information required for creating a certificate including the public key (but not the private key). They're presented to a certificate authority (e.g. a well-known certificate issuer) which can then validate that everything is in order (e.g. if someone requests a certificate for example.com the certificate issuer should ensure that the requestor actually controls that domain), and use its own private key to create a signed certificate from the information contained in the CSR.

Think CSR = certificate - issuer signature.

A certificate contains many pieces of information with the following being of most practical importance:

  • Subject: Who is the owner of the certificate?
  • Subject Public key: Used for communicating with the certificate owner.
  • Issuer: Who signed this certificate?

Certificates are usually not secret since they contain no private information, however, they're sometimes bundled together with their private keys, so care must be taken.

TLS certificates contain different pieces of information, you can look at the contents of a certificate using openssl. When working with certificates it's very likely that you will encounter the following parts:

Information about the certificate subject, who does this certificate belong to?

The Common Name (or CN) is the most basic piece of identifying information about the subject of a certificate and you may well encounter certificates that provide only CNs and no further subject information. Previously the common name was used to verify that a certificate was used for the correct host, so in most cases, certificates still use a hostname for the common name (as seen above).

CN validation may still work with legacy applications but current browsers and libraries are no longer using the common name for validation, instead, they use the Subject Alternative Name (or SAN).

Using SAN has the added benefit of working with a list of domain names and that you may also include IP addresses.

Issuer information about the certificate issuer, which authority signed this certificate?

$ openssl x509 -in example.com.pem -noout -text | grep 'Issuer:'
Issuer: C = US, O = DigiCert Inc, CN = DigiCert TLS RSA SHA256 2020 CA1

It also contains the Authority Key Identifier which allows you to identify the key pair used for signing this certificate.

$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Authority Key Identifier'
X509v3 Authority Key Identifier:
keyid:B7:6B:A2:EA:A8:AA:84:8C:79:EA:B4:DA:0F:98:B2:C5:95:76:B9:F4

Formats of certificates

Common formats for TLS certificates are

  • DER encoded binary data (.der, .cer, .crt)
  • PEM: Base64 encoded DER (.pem but also often .cer and .crt), files begin and end with -----BEGIN CERTIFICATE-----/-----END CERTIFICATE-----. Used by most *nix applications/libraries.
  • PKCS12: Binary data, may include private key, optionally password protected (.pfx, .p12).

Reasons why a certificate may be invalid

As a developer, you're most likely to get into certificates when something doesn't work. Usually, this is because a certificate can't be validated. Here are some of the most likely reasons why a certificate is invalid.

Expired or not yet valid

Certificates are only valid for a specific time window, so not only can they expire they may also be not valid yet.

# Lookup certificate validity for a local certificate.
$ openssl x509 -in example.com.pem -noout -text | grep -A2 'Validity'
Validity
  Not Before: Nov 24 00:00:00 2020 GMT
  Not After : Dec 25 23:59:59 2021 GMT

Some clients may also reject certificates that are valid for too long periods (typically 1 or 2 years), e.g. Safari does not trust certificates that are valid for more than 1 year plus a grace period of 30 days.

Let's Encrypt certificates are also only valid for 90 days which is why they provide tools for automated renewals.

Name validation fails

TLS certificates are only valid for a specific set of domains/addresses which are specified in a certificate's Subject Alternate Name (SAN) field.

# Looking up SAN of a local certificate
$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Subject Alternative Name:'
X509v3 Subject Alternative Name:
  DNS:www.example.org, DNS:example.com, DNS:example.edu, DNS:example.net, DNS:example.org, DNS:www.example.com, DNS:www.example.edu, DNS:www.example.net

The field contains a list of DNS names and/or IP addresses that this certificate is valid for. DNS names may also contain wildcards like *.example.com but note that this does not cover additional subdomains like foo.bar.example.com or the naked domain name example.com.

Since SAN was not always present in certificates you may encounter legacy applications that still rely on validating against the Common Name (CN) of a certificate, which is why this is usually set to the domain name as well.

# Looking at the subject information of a local certificate
$ openssl x509 -in example.com.pem -noout -text | grep 'Subject:'
Subject: C = US, ST = California, L = Los Angeles, O = Internet Corporation for Assigned Names and Numbers, CN = www.example.org

When working with recent applications and libraries setting the common name is not enough and you definitely need to specify SAN.

No trust

Certificates are not trusted automatically but rely on a big bundle of trusted certificate authorities (CA) that are usually distributed as part of an operating system or firmware. Your operating system or device is configured to trust all certificates signed by these CAs. If you encounter a certificate that is signed by a different CA it will be rejected.

Applications may also decide to use their own sets of trusted certificates, e.g. the Java Keystore.

If a certificate is not trusted even though it has been issued by a well-trusted issuer there may be missing intermediate certificates.

Consider the following example: a trusted CA A has issued a certificate to another CA B which allows them to issue certificates of their own. The certificate you're seeing has been issued by B and since B is not included in your trusted certificates the certificate is rejected. To avoid this issue everyone using certificates issued by B should also include the certificate which verifies that B may issue certificates (issued by A). This is an intermediate certificate that allows clients to verify that there is an intact chain of trust from A to B to the certificate they're actually concerned with.

It's of course also possible to encounter certificates from CAs that are simply not trusted e.g. because they're only used for private or internal purposes. If you are certain that such a CA should be trusted you can of course add them to your locally trusted CAs.

Forbidden usage

Certificates can be restricted in their usage, e.g. when you receive a signed certificate from a well-known issuer you're not allowed to sign other certificates, i.e. you may not act as a CA yourself because then you could sign any certificates you wanted to without oversight.

# Checking allowed usage
$ openssl x509 -in example.com.pem -noout -text | grep -A1 'Usage'
X509v3 Key Usage: critical
  Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
  TLS Web Server Authentication, TLS Web Client Authentication

Above you see the typical set of allowed usages for a TLS certificate used for HTTPS, note that certificate signing is not included.

Self-signed certificates

Self-signed certificates are frequently used for testing purposes or in ad-hoc situations. They're signed by the same key that is used for the certificate itself. This implies that the certificate metadata has not been verified by an external entity which is why they're considered not trustworthy.


To learn more about the meshcloud platform, please get in touch with our sales team or book a demo with one of our product experts. We're looking forward to getting in touch with you.