Boundary KMS (Key Management Service) Root Key Migration

HashiCorp Boundary 0.11.1 includes two important new features: KMS key lifecycle management, which enables rotation of internal keys, and the previous-root KMS key purpose, which allows teams to specify a KMS root key used only for decryption. As we dive deeper into each of these features, you will also learn a bit more about how encryption works in Boundary.

»KMS Providers

When you configure your Boundary controller, you have to pick a KMS provider. Boundary supports many different KMS providers, including AWS KMS, Azure Key Vault, GCP Cloud KMS, and of course HashiCorp Vault. The KMS provider provides the root of trust for the keys used for various encryption operations inside Boundary, like encrypting sensitive/secret data stored in the Boundary database, or encrypting the data used to authenticate a KMS worker to a controller.

In addition to choosing a provider, the Boundary KMS subsystem requires you to assign a purpose to a KMS stanza. The currently supported purposes for external KMS providers are:

•  worker-auth
•  worker-auth-storage
•  root
•  previous-root
•  recovery
•  config

This post will explore the root and previous-root purposes. If you want to learn about the other purposes, please consult the full Boundary data security documentation. The root purpose defines the KMS provider that is used for the root key in Boundary.

»KEKs, DEKs, and Key Versions

The root key provides the root of trust for all scope-specific encrypted data in the Boundary database. This works by having several layers of encryption that all link back to the root key. Whenever you create a new scope, Boundary immediately creates one key-encryption-key (KEK), several data-encryption-keys (DEKs), and a new key version for each of the keys. The key version holds the keying material for the key. Using key versions allows you to rotate keying material, while retaining the same key resource. More on that later.

DEKs are used to encrypt sensitive/secret application data in the database. Boundary encrypts the scope DEKs with the scope KEK, and then encrypts the scope KEK with the root key.This way, only the root of trust (the root key) can be used to decrypt the data in the database, since you need to decrypt the KEK first, which can then be used to decrypt the DEKs, which in turn can be used to decrypt the application data.

»Rotating Keys

One of the features introduced with the KMS key lifecycle management in version 0.11.1 is the ability to rotate keys in a scope. When rotating the keys, Boundary simply creates a new key version for the KEK and all DEKs in the specified scope. These new key versions will be used for future encryption operations, while the older key versions may still be used for decrypting existing data in the database. It is considered a best practice to regularly rotate encryption keys to lessen the impact of key compromise, among other factors.

In addition to creating a new key version for all keys in the scope, you can optionally use the rotate key command to rewrap existing key versions with the new KEK version. This means that in addition to encrypting the new DEK versions with the new KEK version, existing DEK versions get re-encrypted with the new KEK version. This leaves all DEK versions, both new and old, encrypted using the new KEK version. This is very useful when you want to stop using an old KEK version.

»KMS Root Key Migration

There are many reasons to consider migrating from one root key to another. Perhaps the old AWS KMS configuration was set up with an account that you no longer want to use, or you want to migrate to a new KMS provider completely (perhaps selecting a cloud agnostic solution like Vault?). Before the 0.11.1 release, it was not possible to migrate KMS providers, but now you have all the puzzle pieces in place to perform a migration.

The first step is to update the existing root purpose KMS stanza to previous-root. This tells Boundary to use this KMS-provided key for decrypting the existing data in the database. For example, here’s what it could look like if you were using an AEAD KMS provider:

 kms "aead" {
-  purpose   = "root"
+  purpose   = "previous-root"
   aead_type = "aes-gcm"
   key   	 = "sP1fnF5Xz85RrXyELHFeZg9Ad2qt4Z4bgNHVGtD6ung="
   key_id	 = "global_root"
}

Next, you add a new root purpose KMS stanza with the new KMS provider configuration and restart the controller. Boundary will immediately start using the new KMS provider for any scopes that are created after the controller is restarted, but the old scopes will still contain KEKs that are encrypted with the previous root key.

+kms "aead" {
+  purpose   = "root"
+  aead_type = "aes-gcm"
+  key   	 = "sF1fnF5Xz85RrXyELHFeZg9Ad2qt4Z4bgNHVGtD6ung="
+  key_id	 = "new_global_root"
+}
+

To fix this, you simply rotate the keys in all the old scopes, making sure to specify the rewrap option, which will ensure that all the KEKs are encrypted with the new root key.

$ boundary scopes rotate-keys -scope-id global -rewrap
$ boundary scopes rotate-keys -scope-id o_B4jfDjZ9jf -rewrap
$ boundary scopes rotate-keys -scope-id p_A4jfDjZ9jf -rewrap

If you want to, you can now remove the previous-root purpose KMS stanza from the configuration file and restart Boundary again. You will know immediately if the operation was successful, as there will be no decryption errors in the log. You’ve successfully migrated from one KMS provider to another, while learning more about how encryption works in Boundary.

»Next Steps

To learn more about how to get started with Boundary please visit our OSS Quick Start Guide for step-by-step tutorials.