Why We Need Dynamic Secrets

Secret management is one of the core use cases for Vault. Today, many organizations have credentials hard coded in source code, littered throughout configuration files and configuration management tools, and stored in plaintext in version control, wikis, and shared volumes. Vault provides a central place to store these credentials, ensuring they are encrypted, access is audit logged, and exposed only to authorized clients.

Achieving this centralization is a huge improvement in security posture, but its not the end of the journey. This is because applications don't keep secrets! It turns out, most applications do a worse job keeping secrets than our close friends. Applications frequently log configuration, leaving them in log files or centralized logging systems. Often secrets will be captured in exception tracebacks or crash reports sent to external monitoring systems, or they will be leaked via debugging endpoints and diagnostic pages after hitting an error. The list of ways applications leak secrets goes on, but the point is applications should not be treated as perfectly secure.

The second challenge after centralization is that services are often sharing the same credentials. This has been a standard practice for applications, but most security organizations would have a panic attack if it was proposed that all human users share the same login credentials. If those credentials leak it is very challenging to identify the source of the compromise or leak. Updating or rotating credentials creates a coordination problem, as there are many users of the credential. Revoking the credential has a large blast radius, potentially taking down multiple service tiers at once.

»Dynamic Secrets

Vault presents an answer to these problems in the form of "dynamic secrets". A dynamic secret is generated on demand and is unique to a client, instead of a static secret, which is defined ahead of time and shared. Vault associates each dynamic secret with a lease and automatically destroys the credentials when the lease expires.

This is an overview of how this works at a high level:

In this example, a client is requesting a database credential. Vault connects to the database with a private, root level credential and creates a new username and password. This new set of credentials are provided back to the client with a lease of 7 days. A week later, Vault will connect to the database with its privileged credentials and delete the newly created username.

Vault supports dynamic secrets with a wide range of systems and is easily extensible with plugins:

• Databases (PostgreSQL, MySQL, MSSQL, Oracle, etc)
• NoSQL (Cassandra, MongoDB, etc)
• Messaging Queues (RabbitMQ, etc)
• Cloud Providers
• SSH
• PKI
• etc.

SSH commonly suffers from having a single widely shared key protecting a large fleet of machines. Vault applies the dynamic secret approach and generates a unique client credential on demand. Vault can either generate One-Time-Passwords (OTP), dynamic RSA keys, or act as a signing authority and use a public key approach.

Using dynamic secrets means we don’t have to be concerned when a developer or operator leaves the organization because things like PEMs won't be shared and will be easily revokable. It also gives us a better break glass procedure should these credentials leak, as the credentials are localized to an individual resource, reducing the attack vector, and the credentials are also issued with a time-to-live (TTL), meaning that Vault will automatically revoke them after a predetermined duration. In addition to this, by leveraging Vault auth and dynamic secrets, you also gain full access logs directly tieing a SSH session to an individual user.

Generating certificates is often so time-consuming and complex that very long-lived certificates are preferred to avoid the overhead. This means certificates valid for years are floating around in plaintext and without any practical means of implementing revocation. This poses a large security risk, as a compromise of a certificate may grant long term access with no simple mitigations. Vault applies a dynamic secret approach and acts as a signing intermediary to generate short-lived certificates. Some Vault users have certificates that are valid for less than an hour!

Dynamic secrets allow us to manage intentions (e.g. web server needs database access) instead of managing credentials. This allows us to achieve the same end goal while solving major challenges:

• Leaky Applications. Dynamic secrets minimize the impact of leaky applications by ensuring credentials are ephemeral. This reduces the risk of credentials that are logged to disk or otherwise exposed because they will be rotated automatically.
• Non-Repudiation. Unique credentials per client makes it much easier to do forensics and understand the source of compromise. By avoiding sharing of credentials we remove the ambiguity of access.
• Automatic Rotation. By attaching a lease to secrets and enforcing a maximum time to live, secrets are being automatically rotated. This improves your security posture while making it easier to be compliant with security standards that require rotation.
• Practical Revocation. The leasing mechanism of dynamic secrets means Vault knows which secrets each client has. This makes it possible to revoke specific leases in the case of a compromise. Unique credentials limit the blast radius of a revocation, preventing full service outages.

_Note: In mainstream IT terminology there are some nuanced differences between rotated credentials vs dynamic secrets

»Dynamic Secrets in Practice

Many applications read their configuration one time at startup, usually from configuration files or environment variables. This raises the question of how to practically use dynamic secrets, without re-tooling to natively integrate with Vault.

The answer lies in the existing configuration files or environment variables! Instead of defining those statically, they can be populated with a dynamic secret so that it's invisible to the application.

In the example above, consul-template is using a template to determine what secrets are needed from Vault. When the secret is fetched, the credentials are dynamically created and the configuration file is written out. The application is then started by consul-template, and reads its configuration like normal. Ideally the configuration file would be written to a ramdisk to avoid persistent disk. When the lease expires on the dynamic secrets, consul-template will re-render the configuration file and either reload or restart the application.

The same technique can be used for applications that read from environment variables, but instead the envconsul tool would be used. Environment variables cannot be changed once an application is started, so when a lease expires the process must be restarted to pick up the new credentials.

»Conclusion

Many organizations suffer from secret sprawl with credentials littered across many different systems, often times in plaintext with limited access control or auditing capability. Vault provides a solution by centralizing secrets and the authorizations around them. While this centralization is a huge improvement in security, dynamic secrets allow us to go even further. Dynamic generation of unique client credentials allows us to create a moving target for attackers and minimize the risk of exposing credentials, while making it easier to do forensics, large-scale time-based rotation, or revocation.

To learn more, checkout:

• Vault dynamic secrets
• Vault Database secrets engine
• Vault PKI secrets engine
• Vault SSH secrets engine