HashiCorp Vault 0.6.2

We are proud to announce the release of Vault 0.6.2. Vault is a tool for managing secrets. From API keys and encrypting sensitive data to being a complete internal CA, Vault is meant to be a solution for all secret management needs.

This blog post covers two releases: 0.6.1 and 0.6.2, which together comprise a major feature release, plus large numbers of additional improvements and bug fixes.

In Detail:

• AppRole Authentication Backend
• Convergent Encryption in transit
• Request Forwarding and Retrying
• Additional Response Wrapping Endpoints

In Brief:

• Key Usage Control and Chained Intermediate Support in pki
• Flexible Filters in ldap
• MongoDB Secret Backend
• Circonus Metrics Integration
• Response Wrapping for /sys Endpoints and List Operations

Please see the full Vault 0.6.2 CHANGELOG for more details. Additionally, please be sure to read the upgrade information at the end of this post.

As always, a big thanks to our community for their ideas, bug reports, and pull requests.

Read on to learn more about the major new features in Vault 0.6.1/0.6.2.

»AppRole Authentication Backend

The AppRole authentication backend is the successor to (and deprecates) App ID. It provides for the same workflows as App ID while also adding a significant amount of extra functionality and greatly enhanced security.

The App ID backend was intended to provide a way to allow authentication to Vault using two independent identification factors: an app ID and a user ID. An out-of-band trusted process would create these values push them into Vault, then push them independently into a client. The mechanism here depended on the specific setup. As one example, one factor could be baked into a specific AMI; the other factor could be delivered to a specific host via Chef or Puppet. At each stage along the way, one half of the identifying factor might be visible to the transport mechanism, but both together would only be visible to the generating service and the client.

There were a few problems with this approach. One is building a trusted third-party service for coordination; ideally Vault should fulfill the coordination role of mapping one identifier to another. Another is keeping both factors fully private from all eyes except for the trusted service and the client; this is often quite challenging.

The AppRole backend was built from the ground up to take advantage of Vault's current feature set. It supports listing, expiration of entries, and issuing periodic tokens. Most importantly, while it retains the ability to use the App ID workflow (known as the "Push" model in AppRole) by manually specifying a role ID and secret ID, it also introduces a "Pull" model.

In the Pull model, rather than a third party service creating credentials and pushing them into Vault, Vault is the source of truth for credentials. Each role has a role ID, with access to read this role ID ACL'd. Each role also has its own set of secret IDs. An authorised user can make a call against the role to generate and return a secret ID, which when combined with the role ID acts to authenticate the caller to Vault.

Importantly, the calls to fetch both the role ID and secret ID can utilize response wrapping to protect them in transit and to guarantee that the secret that was generated was only ever seen by the intended client; even by the caller of the generation endpoint would be unable to see it.

Additionally, the mapping is generated internally and its lifetime is controlled by Vault. Vault can clean up the secret IDs once they are past their validity period and they can be listed and revoked using accessors, similarly to token accessors.

The end result is a more secure, more flexible, and more manageable machine-oriented authentication backend than App ID, and we think it is a very worthy successor.

»Convergent Encryption in transit

The Transit Secret Backend has gained support for convergent encryption.

Normally, using the encryption mode currently offered (AES-GCM), each encryption operation uses a unique nonce, and multiple encryption operations with the same plaintext will produce different ciphertexts. This is the generally-recommended mode of operation for AES-GCM, because if the same nonce is used for more than one encryption operation, the resulting ciphertext can be used to derive information about the key, and all previous and future ciphertext values are now at risk of disclosure.

However, there are times when it is desirable to have the same ciphertext output when the same plaintext is input: convergent encryption. A real-world example is being able to search encrypted data. Imagine that you have customer credit card transaction data that you want to encrypt, but you still want it to be searchable. In order to keep the number of entities that can access the unencrypted data as low as possible, a possible workflow looks something like this:

1. A customer credit card transaction arrives containing the credit card number (in addition to other information, such as transaction ID)
2. The ingress server has permission to encrypt the data with key K and encrypts the credit card number
3. The ingress server places this information into a search service (e.g. Elasticsearch) which indexes using the encrypted credit card number value

You now have a database filled with credit card transactions containing encrypted credit card information. Now, suppose you want to find all transactions given a credit card number:

4. An application server receives a request from a customer to find all transactions made with their card
5. The application server has permission to both encrypt and decrypt with key K and finds the encrypted value of the customer's credit card number
6. The application server kicks off a search using the encrypted credit card number
7. The application server returns the results to the client, without the credit card number ever being durably stored in plaintext

Convergent encryption in transit makes this workflow possible while retaining security due to its use of the transit backend's key derivation mode and generating AES-GCM nonces based on both the context and the plaintext.

»Request Forwarding and Retrying

Since its inception, Vault has redirected clients making requests to standby nodes in a cluster to the active node. While conceptually simple, this had a few disadvantages, most notably when using load balancers in front.

In the best case scenario, the load balancer may fail requests while it waits for a configuration update with the new active node. With some load balancers, the minimum speed that this can happen is ten seconds, even if the new active node was ready nine seconds earlier and the other standby nodes were aware of this change in active status the entire time. To help combat this scenario, Vault now includes request retrying in the CLI and Go API. A request encountering a 5xx status code will retry a default of 3 times (configurable).

However, in very pathological scenarios, a load balancer could detect that an active node was down and start sending requests to standbys, which would forward back to the load balancer, which would send the requests back to standbys, and so on, in a redirect loop.

Another common problem was requests made via curl. Many users did not realize that curl requires use of the -L flag to follow redirects, and would be confused when they would simply see "blank data" being returned.

Vault now includes request forwarding functionality. When enabled (the default), Vault standby nodes will forward requests to the active node and return responses to calling clients. More information is on the HA Concepts page. However, in brief:

When an active node takes over duty, it creates a private key and self-signed certificate, which it advertises through the cryptographic barrier to standby nodes (as a result, just as with client redirection, only unsealed standby nodes can actually forward requests, since they must be unsealed to read the advertisement). The standby nodes use this information to create a mutually-authenticated TLS connection between them and the active nodes. Once this connection is established, client requests can be serialized, sent to the active node for processing, and returned to the caller.

Request forwarding is enabled by default in 0.6.2 (but can be disabled globally or per-client-request).

»Response-Wrapping Endpoints

Response-wrapping was introduced in Vault 0.6 and provides a mechanism for generating secure workflows. In 0.6.2, several new API endpoints were added that enhance this functionality:

• sys/wrapping/lookup: This endpoint allow you to discover a response-wrapping token's TTL and creation time. It is included in the default policy.
• sys/wrapping/wrap: This endpoint allows you to wrap arbitrary data inside a response-wrapped token, allowing you to easily use response-wrapping for secure (or at least, tamper-evident) transport of arbitrary data. It is included in the default policy.
• sys/wrapping/unwrap: This endpoint unwraps a response-wrapping token, but has several benefits over reading cubbyhole/response directly: it verifies that the token is in fact a response-wrapping tokens, eliminating spoofing avenues; and it returns the original wrapped response as its response, rather than returning a JSON string that then must be parsed. This is the new method of unwrapping used by the official Go API, and in the future it will be the only method allowed. This is included in the default policy.
• sys/wrapping/rewrap: This allows rewrapping a response-wrapped token by writing the wrapped response into a new token with the original token's TTL without divulging the wrapped contents.

These new endpoints provide powerful primitives useful for building secure workflows on top of response wrapping.

»Other Features

There are too many new features and improvements in this release to describe all of them in depth, so a few more are covered below in brief:

• Key Usage Control and Chained Intermediates in pki: A set of default key usages are now placed on issued certificates, however, the set is fully controllable at the role level. This increases compatibility with strict RFC-adhering programs such as OpenVPN. Additionally, when setting an externally-generated CA certificate in the backend, a chain of arbitrary length can be included in the PEM file. This chain is retrievable and is included in the response to client generation/signing commands.
• Flexible Filters in ldap: The method of querying and filtering groups in the LDAP server is now a user-specified filter with reasonable defaults. This drastically increases the ability to interoperate with Active Directory and some other setups.
• MongoDB Secret Backend: There is now a backend allowing credential generation and revocation for MongoDB servers.
• Circonus Metrics Integration: Metrics can now be exported to Circonus.
• Response Wrapping For /sys Endpoints: Most endpoints in /sys can now have their outputs response-wrapped. Additionally, response wrapping now works when running list commands.

There are dozens more improvements, plus bug fixes -- see the CHANGELOG for all of the details!

»Upgrade Details

Vault 0.6.1 and 0.6.2 both have behavioral changes that must be understood before upgrading. As such, we provide both general upgrade instructions as well as a version-specific guides (0.6.1, 0.6.2). Note that the 0.6.1 guide covers upgrading from 0.6 and the 0.6.2 guide covers upgrading from 0.6.1; if upgrading from earlier versions, please take note of the upgrade instructions in the corresponding upgrade guides.

As always, we recommend upgrading and testing this release in an isolated environment. If you experience any issues, please report them on the Vault GitHub issue tracker or post to the Vault mailing list.

We hope you enjoy Vault 0.6.2!