Products & Technology

Writing Terraform for unsupported resources

TerraCurl is a utility Terraform provider that allows engineers to make managed and unmanaged API calls in their Terraform code.

Feb 22 2023Rob Barnes

Codifying your infrastructure with Terraform is a great way to ensure repeatability and immutability of your infrastructure, but what can you do when a Terraform provider does not support a resource that is required? In this blog, you will learn how to use the TerraCurl provider to bring unsupported resources under Terraform management.

»Introduction to TerraCurl

Terraform is a powerful tool for automating the creation and management of infrastructure resources across multiple clouds and platforms, from Azure, AWS, and Google Cloud, to Kubernetes and Nomad. Its ability to do so is dependent on the APIs of the target platforms and cloud providers. For Terraform to be able to interface with multiple target platforms, it uses the concept of a Terraform provider, which is a Terraform-centric wrapper around the target APIs, much like a client library in programming ecosystems. These providers implement methods that allow Terraform to create, read, update, and delete the provider resources.

There are situations where the target API supports a resource which is not yet implemented in the Terraform provider. In some cases, this is by design. For example, the Vault provider purposely does not have a resource to unseal a Vault cluster, but the API exists to do so. Providers are maintained by HashiCorp, our technology partners, or the open source Terraform community and pull requests are welcome; however, while waiting for pull requests to be reviewed and merged, you can still deploy resources with Terraform using a provider called TerraCurl.

TerraCurl is a utility provider I developed that allows you to make managed and unmanaged API calls in your Terraform code. A managed API call contains instructions on which API calls to make on create, and delete operations, and Terraform stores the response to the API call in the state file, all of which is accessible for reference by other resources. For all intents and purposes, a managed API call is no different to any normal Terraform resource.

An unmanaged call operates slightly differently as it makes the instructed API call every time Terraform is run, usually at the beginning of a Terraform run. This is designed for instances where information is required from an API call in order for the rest of the Terraform code to function. An example of this would be to look up Boundary scope ID in order for Terraform to configure a child scope.

The goal of this provider is to help out in situations where the platform-native provider does not support the resource you require and the platform API does. It is not intended to replace platform-native providers, which should always be preferred to TerraCurl.

»Managed API calls

Managed API calls require instructions to create and delete a resource. These instructions include:

The API endpoint
The HTTP method
Any HTTP headers (optional)
The request body (optional)
Expected response codes

All managed calls use the terracurl_request resource. The example below shows this in action:

terraform {
  required_providers {
    terracurl = {
      source = "devops-rob/terracurl"
      version = "1.0.0"
    }
  }
}
 
provider "terracurl" {
}
 
resource "terracurl_request" "token" {
  name           = "rob-token"
 
  # Create instructions for Terraform
  url            = "http://localhost:8200/v1/auth/token/create"
  method         = "POST"
  request_body   = <<EOF
{
  "policies": ["web", "stage"],
  "ttl": "1h",
  "renewable": true,
  "id": "rob"
}
 
 
EOF
 
  headers = {
    X-Vault-Token = "root"
  }
 
  response_codes = [200,204]
 
  # Destroy instructions for Terraform
 
  destroy_url    = "http://localhost:8200/v1/auth/token/revoke"
  destroy_method = "POST"
 
  destroy_headers = {
    X-Vault-Token = "root"
  }
 
  destroy_request_body = <<EOF
{
  "token": "rob"
}
 
EOF
 
  destroy_response_codes = [204]
}

The code example above will create and manage a Vault token with a prespecified ID, which is something not supported in the Vault provider by design.The response is then stored in the state file. If none of the expected response codes are received, the Terraform run will fail, as the API call has likely failed. When terraform destroy is run, the destroy instructions are followed. In this case, it will make an API call to Vault to revoke the token. This use case is a good example of the intended purpose of the TerraCurl provider, because the ability to specify the token ID will likely never be supported in the Vault provider.

Note: Generating Vault tokens with Terraform will store them in plaintext in the state file, which has some security implications that may not be suitable for production environments.

»Reading data from the response

Terraform has a utility function built in called jsondecode that allows you to read and access elements of the JSON response to the API call. Adding the code snippet below to the example above will render the full JSON response.

output "response" {
  value = jsondecode(terracurl_request.token.response)
}

To access a specific element from the response, you can drill down further by specifying the element you would like to read. For example, to output the accessor ID of the token, which is nested in the auth object, you can use the following function in the output:

output "accessor" {
  value = jsondecode(terracurl_request.token.response).auth.accessor
}

The process of reading and accessing elements from the response is much like the process of unmarshalling data in programming languages like Golang.

»Unmanaged API calls

Unmanaged calls defer from managed calls because they are not intended to be used to manage the lifecycle of resources. For this reason, all unmanaged API calls in TerraCurl use the terracurl_request data source.

A good example use case for the data source is looking up Boundary resource IDs. The functionality exists in the Boundary API but does not yet exist in the Boundary provider.

data "terracurl_request" "scope_id" {
  name           = "scope_id"
  url            = "http://127.0.0.1:9200/v1/scopes?filter=%22Rift%22+in+%22%2Fitem%2Fname%22&scope_id=global"
  method         = "GET"
}
 
output "scope_response" {
  value = jsondecode(data.terracurl_request.scope_id.response)
}

The example code above makes an API call to Boundary to list all scopes and filters for a scope named Rift. The output contains the decoded JSON response.

scope_response = {
  "items" = [
    {
      "authorized_actions" = [
        "no-op",
      ]
      "authorized_collection_actions" = {
        "auth-methods" = [
          "list",
        ]
        "scopes" = [
          "list",
        ]
      }
      "description" = "Scope for Rift Engineering"
      "id" = "o_u7vZIrVw3W"
      "name" = "Rift Engineering Department"
      "scope" = {
        "description" = "Global Scope"
        "id" = "global"
        "name" = "global"
        "type" = "global"
      }
      "scope_id" = "global"
      "type" = "org"
    },
  ]
}

Just like before, we can drill down further into this response to access a specific element. For example, the scope ID.

output "scope_response" {
  value = jsondecode(data.terracurl_request.scope_id.response).items.*.id
}

»Authenticated and mutually authenticated calls with TLS

The latest release of TerraCurl has the ability to optionally include TLS materials in API calls, as well as verify the server's identity (mTLS)). This is supported in both managed and unmanaged API calls.

The example below shows this in action with an unmanaged API call to a Vault server that requires TLS. Notice the cert_file, key_file, ca_cert_file, and skip_tls_verify arguments in the example.

data "terracurl_request" "test" {
  name   = "vault_seal_status"
  url    = "https://localhost:8200/v1/sys/seal-status"
  method = "GET"
 
  cert_file       = "server-vault-0.pem"
  key_file        = "server-vault-0-key.pem"
  ca_cert_file    = "vault-server-ca.pem"
  skip_tls_verify = false
 
  response_codes = [
    "200"
  ]
}

The same can also be done for managed calls. However, TLS materials are individually scoped between creation and destruction API calls, each of which are optional. For example, you could provide TLS materials for creation calls and omit them for destruction calls. This will depend on how your target API works. In most cases, if TLS is required for creation, it will likely be required for destruction. This will also allow the use of different TLS materials for creation and destruction calls if required.

resource "terracurl_request" "mount" {
  name           = "vault-mount"
  url            = "https://localhost:8200/v1/sys/mounts/aws"
  method         = "POST"
  request_body   = <<EOF
{
  "type": "aws",
  "config": {
    "force_no_cache": true
  }
}
 
EOF
 
  headers = {
    X-Vault-Token = "s.dXHglIuimE3ma89OSjSQpOhy"
  }
 
  cert_file       = "server-vault-0.pem"
  key_file        = "server-vault-0-key.pem"
  ca_cert_file    = "vault-server-ca.pem"
  skip_tls_verify = false
 
 
  response_codes = [200,204]
 
  destroy_url    = "https://localhost:8200/v1/sys/mounts/aws"
  destroy_method = "DELETE"
 
  destroy_headers = {
    X-Vault-Token = "s.dXHglIuimE3ma89OSjSQpOhy"
  }
 
  destroy_cert_file       = "server-vault-0.pem"
  destroy_key_file        = "server-vault-0-key.pem"
  destroy_ca_cert_file    = "vault-server-ca.pem"
  destroy_skip_tls_verify = false
 
 
  destroy_response_codes = [204]
}

»Error handling and retry logic

When interacting with target APIs, there are many reasons why initial calls may fail and subsequent calls succeed. This could be due to a blip in network connectivity or it could be that the behavior of an API is eventually consistent. Whatever the reason is, if there are any errors in the API call, or the expected response code is not received, the Terraform run will fail.

The latest release of TerraCurl introduces retry logic to cater for these scenarios. This feature isn't enabled by default; however, if you do need TerraCurl to retry failed API calls, this can easily be done by adding the retry arguments to the Terraform code. This is possible in both the resource and the data source.

data "terracurl_request" "test" {
  name   = "products"
  url    = "https://localhost:5200/v1/sys/seal-status"
  method = "GET"
 
  cert_file       = "server-vault-0.pem"
  key_file        = "server-vault-0-key.pem"
  ca_cert_file    = "vault-server-ca.pem"
  skip_tls_verify = false
 
  response_codes = [
    200
  ]
 
  max_retry      = 3
  retry_interval = 10
 
}

The code example above shows the retry logic in action using the unmanaged data source. If the initial API call returns an error, or the expected response code is not returned, TerraCurl will wait 10 seconds and re-attempt the call. This will be repeated up to 3 times (not including the initial API call).

As data sources are called at the beginning of Terraform runs, this data source will ensure that Vault is unsealed and ready to accept connections before Terraform attempts to create any resources in Vault. If after 4 attempts (including the initial attempt), there are still failures or a 200 response code is not returned from Vault, the Terraform run will fail.

The code example below shows the retry logic in use with the TerraCurl managed resource. The main difference here is the additional destroy_max_retry and destroy_retry_interval arguments to enable and configure the retry logic on the destroy calls.

resource "terracurl_request" "mount" {
  name           = "vault-mount"
  url            = "https://localhost:8200/v1/sys/mounts/aws"
  method         = "POST"
  request_body   = <<EOF
{
  "type": "aws",
  "config": {
    "force_no_cache": true
  }
}
 
EOF
 
  headers = {
    X-Vault-Token = "s.dXHglIuimE3ma89OSjSQpOhy"
  }
 
  cert_file       = "server-vault-0.pem"
  key_file        = "server-vault-0-key.pem"
  ca_cert_file    = "vault-server-ca.pem"
  skip_tls_verify = false
 
 
  response_codes = [200,204]
  max_retry      = 3
  retry_interval = 10
 
 
  destroy_url    = "https://localhost:8200/v1/sys/mounts/aws"
  destroy_method = "DELETE"
 
  destroy_headers = {
    X-Vault-Token = "s.dXHglIuimE3ma89OSjSQpOhy"
  }
 
  destroy_cert_file       = "server-vault-0.pem"
  destroy_key_file        = "server-vault-0-key.pem"
  destroy_ca_cert_file    = "vault-server-ca.pem"
  destroy_skip_tls_verify = false
 
 
  destroy_response_codes = [204]
  destroy_max_retry      = 3
  destroy_retry_interval = 10
}

»Summary and next steps

This blog post has examined the challenges that engineers sometimes encounter when developing Terraform modules, specifically unsupported resources in Terraform providers. TerraCurl is a flexible Terraform utility provider that can assist in these scenarios and open more possibilities and use cases in workflows.

I hope you have this blog useful and please open a github issue if you have any feedback or feature requests for the provider.

Writing Terraform for unsupported resources

»Introduction to TerraCurl

»Managed API calls

»Reading data from the response

»Unmanaged API calls

»Authenticated and mutually authenticated calls with TLS

»Error handling and retry logic

»Summary and next steps

Sign up for the latest HashiCorp news

More blog posts like this one

New Terraform integrations with Crowdstrike, Datadog, JFrog, Red Hat, and more

Terraform delivers launch-day support for Amazon S3 Tables, EKS Hybrid Nodes, and more at re:Invent

HashiCorp at re:Invent 2024: Infrastructure Lifecycle Management with AWS