Terraform Sentinel v2 Imports Now in Technology Preview

We are proud to announce that the next generation of Sentinel imports for Terraform Cloud are almost here! This is a completely re-designed import API meant to better reflect Terraform 0.12's updated data model, addressing several shortcomings in the existing Terraform 0.11 API and the 0.12 shim.

To get some context around why we've released a new version of the imports, let's discuss the history of the old Sentinel imports, some of the issues that have arisen from their use, and how the new imports, along with recent changes in the Sentinel runtime, have been designed to address some of these shortcomings.

»The Old Sentinel Imports (v0/v1)

Terraform Cloud's Sentinel imports were originally designed around the lower-level details of a Terraform plan. The imports hooked directly into the raw configuration, binary plan, and lower-level JSON state—with most of this data being provided by the singular binary plan data for a particular run within Terraform Cloud.

While this made implementing the imports relatively straightforward, the user experience has been a bit more complicated and exposed certain shortcomings of Terraform's pre-0.12 internals, such as:

• All diff data as strings, regardless of the actual value type. This means that even if the attribute you were looking for was a number or boolean, it will still show up as a string. Casting to the appropriate type is necessary to avoid incorrect matches and runtime errors.
• Diff as a flat, single-tiered map. This meant that you can't use selectors to naturally select a particular field, example: null_resource.foo.diff.triggers.foo is "bar" does not work, and instead needs to be written as null_resource.foo.diff["triggers.foo"] is "bar".
• All state attributes as strings. Similar to the issues with the diff, values found in tfstate - and applied in a resource namespace in tfplan - are all stored as string values, giving you comparison issues similar to the above.

Additionally, due to mostly following convention with how modules and data was laid out in the binary plan, module traversal has historically been done with module traversal functions. These ultimately need to be combined with other functions in order to effectively extract subject resources, creating significant boilerplate (examples can be seen in the example policies found in our terraform-guides repository).

When Terraform 0.12 was released, an all-new data exchange format, decoupled from the lower-level binary internals, was created—the JSON output format, which can be fetched from terraform show -json. Reflective of Terraform 0.12's much richer data internals, these issues have been addressed, in addition to plenty of others. However, in the spirit of backwards compatibility, we ultimately shimmed this new data format on to the existing Sentinel API, meaning that none of these benefits have been able to be realized—until now.

»Introducing the V2 Sentinel Imports

If your workspace is running a version of Terraform 0.12, you can start using the new Sentinel imports now by appending /v2 to your import path, so tfconfig/v2, tfplan/v2, and tfstate/v2.

Note that these imports are in technology preview and are not yet fully stable. This will change soon, and eventually these imports will be the default import in use, and you will have to import the old ones as tfconfig/v1, tfplan/v1, and tfstate/v1. This plan may specifically be tied to your workspace's Terraform version, so it will be important to note for your specific organization's policy sets. Note that the /v1 import paths are also available now, so if you want to future-proof your existing v1 policies for backwards compatibility, you can update the imports immediately.

Let's take a look at some of the changes that the v2 imports are bringing.

»Easy Resource Discovery With filter

One of the most important changes is the restructuring of the data layout into a series of collections in a way that is easily traversed with the recently added Sentinel filter expression.

This is a direct response to the large amount of boilerplate we were seeing in policies that were designed for both resource discovery and attribute filtering.

Let's see what we mean by taking a look at one of our published example policies where we restrict the owners that can be selected by an aws_ami data source. As can be seen in the policy, there is a lot of code that is required for not only module traversal, but also address construction and filtering of resources based on attribute.

This code can now be significantly simplified. With some slight modifications, the entirety of the discovery can be fit into a single filter expression, giving us this policy, complete with preservation of the reporting of violations:

    import "tfstate/v2" as tfstate
    import "strings"

    allowed_owners = [
    	"099720109477",
    	"099720109478",
    ]

    violations = filter tfstate.resources as _, r {
    	r.mode is "data" and
    		r.type is "aws_ami" and
    		r.values.owner_id not in allowed_users
    }

    for violations as _, v {
    	print("data source", v.address, 
        "has unauthorized AMI owner", v.values.owner_id)
    }

    # Main Rule
    main = rule { length(violations) == 0 }

»Semantic Changes

The diff expressed in the resource_changes collection (and to a lesser extent, output_changes) has been re-designed to more closely match the data expressed in the JSON output format. This format is much more expressive than the pre-0.12 diff data format.

First off, the type of change is now expressed as a list of operations, versus a set of flags. This makes it easier to filter out specific kinds of changes, and identify certain operations, such as create-before-destroy operations. The below filter would, for example, filter out all resources that are being created, regardless of if they are being replaced, or created new:

    instances = filter tfplan.resource_changes as _, r {
    	r.mode is "resource" and
    		r.type is "aws_instance" and
    		r.change.actions contains "create"
    }

At the same time, this filter will select instances that are only being created, but not replaced:

    instances = filter tfplan.resource_changes as _, r {
    	r.mode is "resource" and
    		r.type is "aws_instance" and
    		r.change.actions is ["create"]
    }

Additionally, the new change structure addresses the long-running issues relating to structure of changes and typing of changes. Rather than a flat map of diff sets, attributes are now organized into the before and after values within the change, and are fully typed, meaning that you will now see numbers and booleans be represented by their correct types, and lists and maps will be properly expanded. An expression such as the following will now work properly:

    # where r is a resource change for a null_resource
    # formerly r.diff["triggers.foo"].new is "bar"
    r.change.after.triggers.foo is "bar"

    # where r is a resource change for a vsphere_virtual_machine
    # formerly r.diff["memory"].new is "1024"
    r.change.after.memory is 1024

»Raw Data

We also know that some advanced uses of Sentinel may call for having access to the raw data produced by the JSON output, or there might be scenarios that the higher-level collections might not cover.

This is why you can now access all of the JSON plan data under the raw key in tfplan/v2. This data is presented as a map, so no additional processing is necessary to access the fields via Sentinel itself. However, should you need to send it to an external endpoint, you can re-serialize it by using the json standard import.

»Getting Started

For more information on the new imports, check out these pages within the Terraform Cloud documentation:

• tfconfig/v2
• tfplan/v2
• tfstate/v2

As a reminder, the imports are in a technology preview, and the API may be subject to change. We will be issuing another announcement when these imports are stable.

Sentinel is offered with Terraform Cloud's Governance Tier and with Terraform Enterprise. For more information, you can compare plans. To get started with Terraform Cloud (without Sentinel), you can sign up for a free account. If you are interested in hearing more about Sentinel, you can ask our sales team.

Finally, if you are an existing TFC Governance or TFE customer, and you have feedback regarding the new imports, submit a support ticket and let us know!