HashiCorp Vault 0.2

We are proud to announce the release of Vault 0.2. Vault is a tool for managing secrets. From storing credentials and API keys to encrypting sensitive data, Vault is meant to be a solution for all secret management needs.

The initial public release of Vault was almost two months ago and we have been busy extending the core capabilities, adding new secret and storage backends, improving user experience, and fixing bugs.

Vault 0.2 brings many new features including key rotation, rekeying, a PKI secret backend for dynamic certificate generation, a Cassandra secret backend, many new storage backends, and derived unique per-transaction keys for the transit backend. There are so many amazing changes that we cannot possibly list them here, so please see the full Vault 0.2 CHANGELOG for more details.

You can download Vault 0.2 from the project website.

Read on to learn more about the major new features in Vault 0.2.

Key Rotation and Rekeying

When a Vault server is first started, the vault init command is used to initialize the system. This creates the encryption key which is used to protect data at rest, the master key which is used to protect the encryption key, and key shares which are used to split the master key.

In Vault 0.1, all of those were static and could not be changed after they were created. Vault 0.2 allows these to all be updated. This is done by adding two new commands: rotate and rekey.

The first command rotate is used to change the encryption key that is used to protect data at rest. When invoked (either through the CLI or directly with the API), a new random key is generated and installed in the keyring. All new data that is persisted uses the new key, while keys encrypted under older keys can still be decrypted. As of Vault 0.2, the old keys are not automatically re-encrypted but this is a planned future enhancement.

$ vault key-status
Key Term: 1
Installation Time: 2015-07-14 20:46:31 +1000 AEST

$ vault rotate
Key Term: 2
Installation Time: 2015-07-14 20:46:36 +1000 AEST

$ vault key-status
Key Term: 2
Installation Time: 2015-07-14 20:46:36 +1000 AEST

The second command rekey is used to change the master key or the parameters of the key shares. The init command allows the number of shares and the threshold required to be set. Using the rekey command allows those parameters to be changed. This also provides a way to change the key shares if a previous key holder leaves an organization.

The rekey process is a bit more complex, as it must involve the existing key holders. The process is started by specifying the new number of shares and threshold (which could be the same). Once the existing threshold is met, the master key will be regenerated and the new key shares provided.

$ vault rekey -init
Started: true
Key Shares: 5
Key Threshold: 3
Rekey Progress: 0
Required Keys: 1

$ vault rekey
Rekey already in progress
Key Shares: 5
Key Threshold: 3

Key (will be hidden):
Key 1: 9de4b5732ad06ca5c982ebc189d100763e0dfa88afc44ebfe2d707237cde17c001
Key 2: 54cc5de89f2731e46eaac690b3b017fa6173acab9375285bfb360c6b6c6bc15902
Key 3: 2f216b05195ae4b454f98c23ed0aa9a966cb4e187f54a71a1253ab975b0f1d6403
Key 4: 9288da985a7956255b7e8ee75220e27fcaa0dc99b4bb953314822aeb3a674dfb04
Key 5: e965ec75dc048375612dc4540c9a5c2ccd183e2a589a1a72fde78d170d0391c605

Vault rekeyed with 5 keys and a key threshold of 3. Please
securely distribute the above keys. When the Vault is re-sealed,
restarted, or stopped, you must provide at least 3 of these keys
to unseal it again.

Vault does not store the master key. Without at least 3 keys,
your Vault will remain permanently sealed.

Both the rekey and rotate commands can be done online without any disruption to service. They also work in HA environments with multiple Vault instances.

PKI Secret Backend

The new Vault pki secret backend can be used to turn Vault into an internal certificate authority. The backend makes use of the dynamic secret features of Vault to generate x509 certificates on demand for clients.

Running an internal PKI setup has typically been a challenging process due to the automation required to generate certificates, distribute them, and properly manage updating and revocation. This has caused many organizations to issue very long-lived certificates and distribute them with configuration management systems. This creates a large exposure surface and in the case of a compromise the long-lived certificates tend to be usable due to limited use of CRL and OCSP.

The pki backend to Vault makes this process much simpler and more secure. Operators can simply provide Vault with a root or intermediary certificate and define various roles which require certificates. Clients of Vault can requests certificates as needed, and have them generated on demand. Instead of using very long-lived certificates, Vault can generate certificates that last only minutes, and instead are frequently updated. This limits the exposure in case of a compromise, and combined with automatic CRL generation makes it much safer to issue certificates.

$ vault write pki/issue/common common_name=www\.hashicorp\.com
Key             Value
lease_id        pki/issue/common/819393b5-e1a1-9efd-b72f-4dc3a1972e31
lease_duration  259200
lease_renewable false
certificate     -----BEGIN CERTIFICATE-----
MIIECDCCAvKgAwIBAgIUXmLrLkTdBIOOIYg2/BXO7docKfUwCwYJKoZIhvcNAQEL
...
az3gfwlOqVTdgi/ZVAtIzhSEJ0OY136bq4NOaw==
-----END CERTIFICATE-----
issuing_ca      -----BEGIN CERTIFICATE-----
MIIDUTCCAjmgAwIBAgIJAKM+z4MSfw2mMA0GCSqGSIb3DQEBCwUAMBsxGTAXBgNV
...
-----END CERTIFICATE-----
private_key     -----BEGIN RSA PRIVATE KEY-----
MIIEowIBAAKCAQEA0cczc7Y2yIu7aD/IaDi23Io+tvvDS9XaXXDUFW1kqd58P83r
...
3xhCNnZ3CMQaM2I48sloVK/XoikMLb5MZwOUQn/V+TrhWP4Lu7qD
-----END RSA PRIVATE KEY-----
serial          5e:62:eb:2e:44:dd:04:83:8e:21:88:36:fc:15:ce:ed:da:1c:29:f5

The pki backend is another example of how Vault can be used as a security foundation, to go beyond just storing secret data. Using the pki backend makes it simpler for organizations to embrace mutual TLS and begin to migrate to a zero trust datacenter.

A major thanks to Jeff Mitchell for contributing this backend and to Akamai for sponsoring his efforts.

Per-Transaction Derived Keys

The transit secret backend has been available since the initial release of Vault. The backend is used to help encrypt and decrypt data in transit through Vault. We previously blogged about how HashiCorp uses Vault's transit backend to protect the sensitive data stored by Atlas.

At the core, the transit backend manages a set of named keys. The keys are held within Vault, and data can be sent in plaintext to be encrypted within Vault or ciphertext sent to be decrypted. This allows clients to manipulate data without ever having access to the encryption keys. This makes it easier to manage PII information in web servers, while still using RDBMS systems for scalable storage.

With Vault 0.2, the transit backend now supports setting the derived property of a named key. When a key is created with this property, any encrypt or decrypt operation must also provide a context along with the plaintext or ciphertext.

This context is used with the raw encryption key to generate a derived key. This transaction key is derived using the secured high entropy key, and is used to encrypt or decrypt the provided data. This allows each transaction using the same named key to use a different encryption key. A compromise of any transaction does not reveal any data about any other transactions with a different context or the raw encryption key.

$ vault write transit/keys/foo derived=true
Success! Data written to: transit/keys/foo

$ CTX=`echo -n foo | base64`
$ vault write transit/encrypt/foo context=$CTX plaintext=`echo -n bar | base64`
Key         Value
ciphertext  vault:v0:DykggPIDCMz+vqrWsqa309HtQs2c3Y2BxedNCTlQQQ==

$ vault write transit/decrypt/foo context=$CTX ciphertext="vault:v0:DykggPIDCMz+vqrWsqa309HtQs2c3Y2BxedNCTlQQQ=="
Key         Value
plaintext   YmFy

$ echo YmFy | base64 -D
bar

The actual implementation of derived keys is based on the NIST SP 800-108 recommendations.

ACL Policy Improvements

The policy language for Vault was inspired by the Consul ACL policy language. In Vault 0.1, the policies looked like:

path "sys/" {
    policy = "read"
}

path "secret/" {
    policy = "write"
}

In this original version, all path stanza's would glob and greedily match on the request path by default. This made it impossible to specify an exact-match policy. With Vault 0.2, this is now made explicit with the * glob wildcard:

path "sys/*" {
    policy = "read"
}

path "secret/*" {
    policy = "write"
}

# Only permit the 'foo' key exactly
path "transit/encrypt/foo" {
    policy = "write"
}

Second, Vault is always default deny system, meaning unless permission is explicitly given it will be denied by the ACLs. However, with Vault 0.1 the policy language gave the deny policy the lowest precedence, this made it hard to black list specific paths. Vault 0.2 now gives it the highest precedence. This enables the following;

path "secret/*" {
    policy = "read"
}

path "secret/super-secret" {
    policy = "deny"
}

With Vault 0.2, any operation to the secret/super-secret path would now be denied where it was previously allowed by the first rule. Please note that this may be a breaking change for existing policy files.

Upgrade Details

Vault 0.2 introduces some major internal changes to support the rekey and rotate commands. The policy language changes similarly change the way policies are specified and enforced.

All of these upgrades however are done online when the new version of Vault is unsealed. The existing representation of the encryption key will be upgraded to the keyring in place. Any existing policies will be updated to ensure the implicit glob behavior continues to work.

All policy files should be upgraded, since any new policies written are assumed to conform to the Vault 0.2 specification. There is no way to use the old specification as an attempt to deprecate it while the project is still young.

Roadmap

Vault 0.2 is a major release that extends the functionality and fixes many major usability and stability issues. As a result, we expect that there will be some new issues which will be address in point released following. We are expecting to start a full external code audit of Vault shortly.

Following that, Vault 0.2 will be focused on adding support for vault ssh, which will use one-time-keys and one-time-passwords for solving the shared SSH key problem, adding support for new secret, auth, and credential backends, improving usability and fixing bugs.

As always, we recommend upgrading and testing this release in an isolated environment. If you experience any issues, please report them on the Vault GitHub issue tracker or post to the Vault mailing list.