Ephemeral values in Terraform

Before Terraform 1.11, working with sensitive values was a challenge — sensitive values were always persisted to the Terraform plan artifact and state. As a result, we advised (and still advise) treating your state as sensitive data. This advice applies equally to us as it does to you. To address this, we introduced the concept of ephemerality in Terraform, allowing you to work with sensitive data like passwords and tokens securely, without leaving a trace.

»Ephemerality

Ephemerality in computing refers to the ability to create something that is short-lived or temporary — a piece of data that exists only for a brief period and is discarded once its purpose is fulfilled. In Terraform, this concept was introduced to manage sensitive data or open a connection in a way that ensures it doesn’t persist beyond its immediate use. We’ve implemented new language constructs that track values only at runtime, making them transient and, therefore, ephemeral by design. These are part of what we call ephemeral values, a term that covers both ephemeral resources, ephemeral input variables, and write-only arguments.

»Ephemeral resources

Ephemeral resources are Terraform resources that are essentially temporary. They are responsible for reading data from a source such as a secrets manager, or opening a connection, and their attributes can be referenced in other places without persisting anything to the Terraform plan artifact or state file.

It’s important to note that ephemeral resources require all their dependencies to exist because they always run during both the plan and apply stages. If an ephemeral resource attempts to read a secret from a secrets manager that doesn’t exist, it will result in an error. However, Terraform can defer the execution of an ephemeral resource to the apply stage if one of its input arguments references a value that is not yet known at the plan stage but will be determined during apply.

Here’s an example of an ephemeral password resource with no dependencies, executed during both plan and apply:

This generates an ephemeral password string using a cryptographic random number generator. The generated string can then be used as input for a write-only attribute on a managed or ephemeral resource.

»Write-only arguments

Write-only arguments are managed resource attributes that are configured by users but are not persisted to the Terraform plan artifact or state file. Terraform providers implement write-only arguments on managed resources to handle sensitive values, such as passwords, tokens, and other secrets, securely.

An example of a write-only argument is the secret_string_wo argument on the aws_secretsmanager_secret_version resource:

It’s common for write-only arguments to be assigned an ephemeral resource attribute, as shown in the example above. This enables end-to-end ephemerality in Terraform, ensuring that sensitive values are not persisted in either the Terraform plan artifact or the state file.

Earlier, we mentioned that ephemeral resources are executed during every plan and apply. This means that a new ephemeral random password is generated, or a new value is fetched from a secrets manager, with each plan and each apply. To prevent a write-only argument from being updated on every run after creation, you set the write-only version argument. Terraform stores this version in state and uses it to track changes. When the version is incremented, Terraform allows the resource to accept the new value for the write-only argument, which is then sent to the provider to update it accordingly.

»Write-only version arguments

Terraform doesn’t store write-only arguments in the plan or state file, meaning it can’t track changes to their values. As a result, write-only arguments are sent to the provider during every operation. However, providers typically include write-only version arguments (e.g. password_wo_version) alongside write-only value arguments (e.g. password_wo), where the version arguments are stored in the state file and can be tracked for changes.

These write-only version arguments enable users to update a write-only value by incrementing the version number. When the write-only version argument is updated, Terraform detects this change in the plan and notifies the provider. The provider then uses the updated write-only value to update the resource accordingly.

If a resource depends on multiple write-only arguments in the dependency graph, it’s a good idea to keep their version arguments in sync. This way, all the related write-only values are updated together, avoiding inconsistencies and ensuring the provider uses the correct values during each operation. In the next section, we’ll cover deferring ephemeral resources, and the example code will demonstrate how to synchronize versioning for write-only arguments.

»Deferring ephemeral resources

If an input argument of an ephemeral resource references a value that is not yet known but will be during or after the plan, Terraform defers the resource’s execution to the apply stage instead of running it during the plan. This behavior allows Terraform to evaluate the ephemeral resource at the correct time and ensures that the resource is not executed prematurely.

Similarly, ephemeral resources form nodes in Terraform’s dependency graph. When a managed resource or data source depends on an attribute of an ephemeral resource, Terraform automatically provisions the ephemeral resource first.

As we concluded earlier, ephemeral resources are executed during every plan and apply. It’s important to model your Terraform configuration in such a way that the dependency graph of all of the managed resources and ephemeral resources ensures correct execution order and prevents unintended behavior.

By using the ephemeral random password and AWS Secrets Manager secret we created earlier, we can take advantage of ephemerality and its deferring logic to securely provision an AWS RDS instance.

In the above example, the ephemeral resource aws_secretsmanager_secret_version depends on an argument that Terraform doesn’t initially know. Terraform defers executing this ephemeral resource until the apply stage, ensuring it runs only after the necessary information becomes available.

During the apply stage Terraform first creates the secret in AWS Secrets Manager using the ephemeral random_password, then Terraform retrieves the secret using the ephemeral aws_secretsmanager_secret_version resource. The last thing Terraform does is write the password to the write-only password_wo argument of the aws_db_instance resource.

»Lifecycle of an ephemeral resource

Ephemeral resources have a lifecycle that differs from other resources and data sources. The lifecycle has three steps:

• Opening: When Terraform needs the result of an ephemeral resource, it opens it — much like reading a secret from a secrets manager.
• Renewing: If Terraform needs access to the ephemeral resource for longer than the remote system’s enforced expiration time, Terraform asks the provider to periodically renew it.
• Closing: Once Terraform no longer needs an ephemeral resource, it closes it. This happens after the providers that depend on an ephemeral resource complete all of their work for the current Terraform run phase.

All ephemeral resources implement a form of opening logic, but not all implement closing or renewing logic. Whether they do depends on the ephemeral nature of the resource — handling tunnel connections, for instance, may require periodic renewal and proper closure of the connection. Leasing a temporary set of credentials from Vault would similarly imply renewal and revocation of those credentials when they're no longer needed.

»Persisting ephemeral secrets

Ephemerality in Terraform prevents introspection of the value itself, which is the expected and intended behavior. However, this could lead to a potential issue. When using the ephemeral random password, you’re generating a new secret that hasn’t been persisted to a secrets manager yet, so it’s important to store it to ensure it remains accessible. This is why, in the third code snippet above, we write the ephemeral random password to a secrets manager first and then fetch it from there during the apply stage using another ephemeral resource, effectively persisting an ephemeral secret to a secrets manager.

However, if you’re reading existing secrets from a secrets manager that’s managed outside of the Terraform module you’re working in, there’s no need to persist them again.

»Learn more about ephemeral values

For more information on working with ephemeral values, resources, and write-only arguments, check out the ephemerality in resources documentation. You can find detailed information on ephemeral resources in the ephemeral resource block documentation, or learn more about write-only arguments in the write-only arguments documentation.