Managing Credentials in Terraform Cloud & Enterprise

UPDATE: Terraform Cloud & Enterprise now have a native workflow for dynamic credential management. For a consistently updated guide on how to handle dynamic secrets in your Terraform workflows using the most secure method, visit Dynamic Provider Credentials for an introduction and then follow the Authenticate providers with dynamic credentials tutorial. Dynamic provider credentials integrate with secrets management workflows using HashiCorp Vault, AWS, Azure, and Google Cloud.

There are several ways to manage credentials or other secret types in Terraform Cloud and Terraform Enterprise, either natively, or with purpose-built secrets management utilities like HashiCorp Vault, so this is a somewhat opinionated article that lists what I believe are currently the best options. Please note this is not a replacement for some other best practices, such as keeping your Terraform CLI and code up to date.

The described patterns follow some common principles. Credentials should be:

• As unique as possible per workspace
• Easy to rotate
• As dynamic as possible
• Protected with RBAC

There are also some nuances between Terraform Cloud and Terraform Enterprise that I will call out in each section.

In terms of security, both Cloud and Enterprise products encrypt their sensitive variables using the Vault Transit secrets engine and do not allow any external API call to decrypt these values. For more details please see the Data Security page.

If you still have questions by the end of this blog post, try attending one of our community office hours or booking a 1:1 technical session.

»Managing Credentials Using Only Terraform Workspaces

Using only Terraform workspaces, your cloud vendor's Terraform provider, and the Terraform Cloud/Enterprise provider, you can set up a “Credentials” workspace that is able to generate new credentials and rotate the ones used by other workspaces. To avoid the secret zero problem, when setting up the credentials workspaces, you can make use of the Terraform Agents pattern described in the next section.

Note: In Terraform Cloud all workspaces live in the same Organization.

Pros

• There’s a separation between development environments.
• It’s possible to use different types of secrets.

Cons

• Requires a 3rd party API trigger to force a re-deployment of all “Credentials” workspaces.
• Complex RBAC setup that does not exist in the free tier.
• Secrets are stored in the “Credentials” workspace state.
• “Credentials” workspaces require a Terraform Cloud/Enterprise User or Team token to access other workspaces that reside in another Organization.

»Managing Credentials Using Only Terraform Agents

Both Terraform Cloud Business tier and Terraform Enterprise support running your code using external agents. This feature is called Terraform Agents. Any cloud provider declared in your Terraform code is able to take advantage of the credentials set in the Terraform Agent environment, which means the credentials do not need to be set at the workspace level.

HashiCorp Solutions Engineer Andy Assareh has a repo and a recording to help walk you through this pattern.

Pros

• There’s a separation between development environments.
• No credentials are set in project workspaces.
• No credentials are stored in the workspace state.
• One of the least complex patterns.

Cons

• Only addresses cloud access credentials, not other kinds of secrets.
• Requires Terraform Cloud Business tier.
• Careful RBAC is required to ensure developers are not able to change Agent Pools in the workspace.
• Difficult to scale and audit when using multiple accounts.
• In Terraform Cloud Business tier, there are some limits on the number of Agents and Agent Pools an Organization can have.
• Difficult to have a 1:1 mapping between workspace and Agent.

»Managing Credentials Using Vault

It’s no secret HashiCorp Vault is able to generate dynamic credentials for all major cloud vendors, databases, etc. For those who know about Vault, this integration with Terraform is the first solution they ask about, since secrets management is Vault’s primary use case. Here are a few patterns to make that integration work.

»Direct Integration with a Vault Plugin

There isn’t much to explain in this workflow. Most of the complexity is in setting up Vault authentication and some Terraform template code.

Pros

• Dynamic credentials are used to access Vault using the workspace identity, meaning there are no hardcoded credentials.
• Different credentials for each environment/workspace.
• Different credentials for planning and applying. For example, speculative plans can have read-only credentials because Vault can create different entities depending on the run status.
• Cloud credentials retrieved from Vault are ephemeral.
• Support for multiple types of Vault secrets.
• Low complexity Terraform code.

Cons

• Need to install a 3rd party plugin. More details are in the plugin repo.
• Because we are using the Vault provider, secrets will be present in the state file.
• Requires Terraform code changes to use the Vault provider.
• Increase in complexity to set up Vault correctly.

»Integration with CI/CD

At the heart of this integration pattern, is the ability to confidently authenticate to Vault with an identity that is unique and combines workflow, repository, and branch/environment.

This is not the case with all CI/CD implementations, so I’m only mentioning two implementations where I know it’s possible.

Vault tokens should be very short-lived and linked to an entity, to restrict access in case any token gets leaked and allow for better auditing. Avoid storing Vault tokens as secrets!

Regarding the workflow, the CI/CD runner will need to authenticate to Vault and retrieve:

• A dynamic TFE/TFC user token
• A dynamic cloud access credential
• Any other secret to be used by the workspace

Pros

• It’s possible to have 0 hardcoded or long-lived credentials, using only identities.
• There’s a separation between development environments.
• Different credentials for different branches.
• Terraform Enterprise and Terraform Cloud credentials are not stored in Terraform state or the CI/CD platform.

Cons

• Requires a CI/CD system able to assign an identity that combines a unique run id, workflow/pipeline, repository, and commit id.
• May lead to secret sprawl.
• Increase in pipeline complexity.

These pros and cons apply to the two subsections below.

»GitLab CI/CD Integration

GitLab runners are able to authenticate to Vault using the JWT auth backend, where you can configure separate roles for staging, dev, or production.

Once authentication has been solved, GitLab CI/CD will be able to retrieve the necessary secrets and interact with the Terraform Cloud/Terraform Enterprise API. Here’s an example.

For private repositories, you’ll need to have GitLab Premium.

»GitHub Actions Integration

For GitHub Actions to authenticate to Vault there are two options:

• Set an AppRole as an Environment Secret in your repo and use the Vault GitHub Action
• Use the community Vault Auth plugin for GitHub Actions

If you use AppRole, please make sure you also have a cron job set up to rotate the environment secrets, and use the “Repository Environments” to distinguish between production and non-production.

If you use the custom Vault GitHub Action authentication backend, right now it’s not possible to distinguish which branch is being executed.

»Pick the Solution that's Right For You

There is no right answer or one-size-fits-all solution to managing your credentials and secrets within Terraform Cloud and Terraform Enterprise. This is highly dependent on your current requirements, environment, tools, etc, so it’s up to you to select the pattern that is best for you and your team, taking into account the pros and cons of each solution.

Please bear in mind that all of these solutions require you to plan and execute an efficient team/application onboarding process to be successful. To learn more about onboarding applications into Vault check out our other blog post Onboarding Applications to Vault Using Terraform: A Practical Guide or book a 1:1 session with one of our experts.