Using HashiCorp's Vault with Chef

One common challenge organizations face when integrating Vault by HashiCorp in their infrastructure is how to fetch secrets from Vault using a configuration management tool. This blog post details a few techniques for retrieving secrets from Vault using Chef, but the topics can be broadly applied to any configuration management software such as Puppet or Ansible.

»Overview

For each of the following techniques, we will look at two different secret-retrieval scenarios with Vault. The first scenario involves retrieving a static secret from the Vault generic secret backend. Here is some sample data from the generic secret backend:

$ vault read secret/vpn
Key             Value
lease_duration  2592000
password        b@c0n
username        sethvargo

Vault's generic secret backend can be thought of as an encrypted key-value store. The vault read command above shows how the data can be structured. We can reference these named pairs when retrieving the secret. Generic secrets do not expire, but they can be revoked at any time.

The second scenario involves retrieving dynamic PostgreSQL credentials from Vault using the Vault PostgreSQL secret backend. Vault is configured with the root credentials to Postgres and then generates unique, leased credentials using those root credentials. Once configured, we can generate new credential sets like this:

$ vault read postgresql/creds/readonly
Key             Value
lease_id        postgresql/creds/readonly/c888a097-b0e2-26a8-b306-fc7c84b98f07
lease_duration  3600
password        34205e88-0de1-68b7-6267-72d8e32c5d3d
username        root-1430162075-7887

When we read this path, Vault will dynamically generate a new set of credentials for connecting to Postgres using the given SQL. The returned credentials have an associated lease, and it is the application's responsibility to renew the credentials before the lease expires or Vault will revoke them.

»Method 1: Reading Secrets at Runtime

Our recommended approach to use Vault with any configuration manage tool is to move the secret retrieval and renewal into a runtime process instead of a build time process. This is especially useful if you are using Chef and Packer to build artifacts or if you do not run Chef on a regular interval. In this case, traditional configuration management is used to install a tool that manages interaction with Vault at runtime.

Consul Template is a single static binary that can retrieve data from Consul, Vault, or both. Despite unfortunate naming, Consul Template does not require you are using Consul. Provide Consul Template with an input template, output destination, and an optional arbitrary command to execute when the template contents are changed. Prepare for a little bit of "meta", because we are going to use a Chef template to render a Consul Template template that will start as a runtime process to render our application's configuration.

Chef's responsibility:

• Install Consul Template binary
• Add Consul Template upstart configuration

Consul Template's responsibility:

• Retrieve secrets from Vault
• Write secrets to disk and manage application configuration lifecycle
• Handle secret lease renewal

This post will only include relevant snippets, but a complete recipe and supporting files is available as a GitHub gist for using Vault with Chef. The Chef recipe downloads Consul Template from the HashiCorp releases service, unzips and extracts it into /usr/local/bin, creates the "dot-d" directory for our Consul Template configurations, and configures upstart to manage the service.

Now that Consul Template is installed and running as a service, we need to create our Consul Template template which will render the application configuration. Since this is a fictitious application, let's assume our application accepts a configuration file like this:

[config]
  username = "..."
  password = "..."

We need to populate the values in ... with secrets from Vault. Here is how we do that using Consul Template's templating language:

{{ with vault "postgresql/creds/readonly" }}
[config]
  username = "{{ .Data.username }}"
  password = "{{ .Data.password }}"
{{ end }}

This template will be vendored with the application, so let's assume it already exists on the system /opt/my-app/config.ctmpl. Next, we have to tell Consul Template to parse this template and output the configuration on the other side. That configuration looks like this:

# cookbook/templates/my-app-ct.hcl
template {
  source = "/opt/my-app/config.ctmpl"
  destination = "/opt/my-app/config"
  command = "service my-app reload"
}

Once this file is placed in /etc/consul-template.d, Consul Template will start reading the template at /opt/my-app/config.ctmpl, rendering the result to /opt/my-app/config, and reloading the service when the contents change.

Whether Chef is used at runtime or build time, the secret management is left entirely to Consul Template.

»Advantages

• Secrets never touch Chef
• Chef does not need to run on a regular interval
• Chef does not need to interact with Vault
• Very Chef-like "download, install, service, template" pattern

»Disadvantages

• Go's templating language
• Added indirection

»Method 2: Reading Secrets at Configuration Time

Another common approach to using Vault with configuration management tools is to populate secrets during the converge stage. For Chef, the easiest way to do this is using the vault-ruby Rubygem, but Vault also has a JSON HTTP API, and many client libraries exist for other languages.

# cookbook/recipes/default.rb
creds = Vault.logical.read("secret/vpn")

template "/etc/vpn/config" do
  source "my-vpn-config.erb"
  owner "vpn"
  group "vpn"
  mode "0600"
  sensitive true
  variables(
    :username => creds.data[:username],
    :password => creds.data[:password],
  )
end

In this example, the Chef run queries the Vault server for the value at "secret/vpn" and passes the values at "username" and "password" as variables into the template.

»Advantages

• Very "Chef-like"
• Minimal barrier to entry
• Works very similar to data bags or encrypted data bags

»Disadvantages

• Unable to handle lease revocation events
• Unable to handle secret updates
• Can easily become stale
• Cannot use secrets that require lease renewal (read on)

While this approach may work well for secrets stored in the generic backend, it will not work with dynamically generated secrets. Here is a relatively similar Chef recipe that pulls credentials from the dynamic Postgresql backend we configured and puts them into a an application configuration:

# cookbook/recipes/default.rb
creds = Vault.logical.read("postgresql/creds/readonly")

template "/opt/my-app/config" do
  source "my-app.conf"
  owner "my-app"
  group "my-app"
  mode "0600"
  sensitive true
  variables(
    :db_username => creds.data[:username],
    :db_password => creds.data[:password],
  )
end

Assuming you are properly authenticated to Vault with permissions to read from that path, the Chef run will complete successfully. However, Chef does not have a mechanism to renew these secrets. Unlike the generic secret backend, the dynamic backends hold a lease that must be renewed before the TTL expires. If the lease for the secret is not renewed before the lease expires, Vault will revoke the secret. Even if the Vault is offline, the credentials were created using Postgres' VALID UNTIL clause, so the database will also revoke the credentials on our behalf without a renewal. This is a difficult bug to try and diagnose, because it will only occur when you have not run Chef in the scheduled interval before the lease expires. If you are not running Chef on a regular interval (maybe you are using Chef with Packer to build AMIs or Docker containers), the secrets will likely expire before you launch the application. This is why we consider secrets to be "runtime" configuration, not "build time" configuration.

Even if you execute Chef on a regular interval, each Chef run will create a new set of credentials. This means the application will receive a fresh set of Postgres credentials on each iteration. Not only will this create many unused secret entries in Vault and Postgres, but it could also complicate diagnosis via Vault's audit log in the event of a compromise. While these entries will be cleaned up by Vault, the application will continuously receive a new set of credentials and Chef will restart the application.

»Method 3: Custom Resource and Provider

If you are comfortable writing Ruby and custom Chef extensions, using a custom resource and provider (or LWRP) is a good technique for retrieving and managing secrets with HashiCorp's Vault and Chef. This technique relies on notifications to control flow. The Chef recipe code looks like this:

# cookbook/recipes/default.rb

vault_secret "postgresql/creds/readonly" do
  notifies :create, "template[/config]", :immediately
end

template "/config" do
  source "my-app.conf"
  owner "my-app"
  group "my-app"
  mode "0600"
  sensitive true
  variables lazy {
    {
      :username => node.run_state["postgresql/creds/readonly"].data[:username],
      :password => node.run_state["postgresql/creds/readonly"].data[:password],
    }
  }
  action :nothing
end

First, you will notice there is a new resource, vault_secret, that accepts a path as the name argument. This resource notifies our template to create (or update if it already exists). The template remains relatively unchanged save for addition of the action :nothing and a slight change to the way secrets are retrieved. Of note is the use of lazy - this is important because the node's run state is not populated until after the secret is retrieved. The full LWRP code can be found in this Gist.

This custom resource fires notifications when a read or renew of a Vault secret occurs. A read should only occur if the secret did not already exist or if the secret lease renewal failed. In both of these scenarios, the template needs to be re-rendered. However, instead of persisting the entire secret into the node object, this LWRP pushes the secret management responsibility into the notified resource. In the case of our template, once the secrets are written to the configuration file, they are no longer needed by Chef. As such, we do not need to store the raw secret.

The LWRP uses the node's run_state to store temporary data for the duration of this Chef run. The secrets only persist in memory and are never stored on the node object that is persisted back to a Chef Server. Because the data is not stored on the run state until after this LWRP executes, we need to use Chef's lazy when setting the template variables.

Lastly, you may have noticed that we are no longer relying on the recipe to provide the path to the lease_id. The LWRP does accepts a property called destination, but the default value is the same as path. If specified, the destination field determines the path where the lease is stored on the node object and path on the node's run state where secrets are stored in memory.

»Advantages

• Still very Chef-like
• Does not store secrets on the node object
• Flexible and extensible to multiple patterns
• Can notify multiple resources if they share the same secrets

»Disadvantages

• Requires custom resource
• Requires Chef Client run on an interval less than the lowest lease

»Method 4: Directly Integrate into Application

The last approach to using Vault with Chef is to bypass configuration management entirely, and add support for Vault directly into your application. Instead of relying on your configuration management tool or Consul Template to communicate with Vault, your application will communicate directly with Vault. This means your application needs to be responsible for retrieving secrets and renewing leases. Since this technique is dependent upon the application's needs, an example is not provided.

»Advantages

• More performant
• More control over interactions with Vault
• Less responsibility on configuration management

»Disadvantages

• Requires applications to be Vault-aware
• Requires engineering effort
• Slower (for existing applications)

»Missing Pieces

This post omitted some key pieces that are helpful to call out. First you must unseal Vault in order to add secrets. Then when applications request access to those secrets, Vault does not give out the root credentials. Instead, it generates unique, leased credentials. The final piece is how applications authenticate with Vault and validate that they are able to access the requested secrets. This is done by giving the application a VAULT_TOKEN, which can be done with an app_id or cubbyhole, for example.

»Conclusion

There are multiple ways to integrate Vault by HashiCorp with Chef, and each solution has advantages and disadvantages. The easiest way to integrate Vault into your configuration management solution is to use Consul Template, but other solutions may be acceptable depending on your use case. We look forward to seeing what kind of amazing integrations you can build with Vault!