HashiCorp Vault Announces Integrated Storage General Availability

We are excited to announce the general availability of the Integrated Storage backend for Vault with support for production workloads. Integrated Storage exists as a purely Vault internal storage option and eliminates the need to manage a separate storage backend. Integrated Storage inherits a number of the benefits from the Consul storage backend and improves the user experience.

In this blog, we will discuss the capabilities of Integrated Storage, as well as its differences from Consul as a storage backend. We will also discuss the factors affecting the decision of whether or not to migrate to Integrated Storage (if using Consul), and provide references to relevant resources such as the new references architecture guide.

»Integrated Storage Capabilities

Integrated Storage is a Vault internal storage option that leverages the Raft consensus protocol to persist data to disk.

Here are a few feature highlights of Integrated Storage:

• Supports Vault Enterprise replication capabilities
• Provides backup and restore capabilities
• Supports High Availability
• Provides automated node to node TLS by using Vault’s cluster communication
• Allows for the join process to be automated with auto-unseal

»User Experience

Integrated Storage eliminates the need to set-up, manage, and monitor a third-party storage system such as Consul, resulting in operational simplicity as well as lower infrastructure cost. When using Integrated Storage, troubleshooting Vault becomes much easier because there is only one system to investigate, whereas when running Vault in addition to an external backend storage system like Consul, you would have to debug two systems and possibly the network in between.

When comparing Integrated Storage to the Consul storage backend, Integrated Storage provides better network performance because there is no additional network hop to Consul. There are differences in how system resources are consumed too. For example, Integrated Storage writes updates to the disk which allows Vault's dataset to not be bound by the amount of RAM on the host, whereas Consul loads the entire data set into RAM. With Integrated Storage, data is on-disk and bound by disk I/O (SSDs are recommended), which results in an extra disk write compared to Consul. Due to these differences, each storage option has its own reference architectures, suggested system requirements (machine specifications, SSDs, network requirements, etc.), performance characteristics, and data inspection methods.

As you can see, the Integrated Storage backend offers many improvements over a non-integrated backend. Still, there are likely to be some operational and performance differences between Integrated Storage and your current backend. Given that, we highly recommend that if you are running in production today, and interested in migrating your storage backend, you create a test environment and explore the Integrated Storage backend with a workload similar to your production environment. The best way to gauge performance is to benchmark in your own environment using your workloads.

We have created Learning Guides to assist you here:

• Vault HA Cluster with Integrated Storage
• Vault with Integrated Storage Reference Architecture
• Preflight Checklist - Migrating to Integrated Storage
• Inspecting Data in Consul Storage

»Should You Migrate from Consul?

As already mentioned, Integrated Storage is an additional storage option made available in Vault 1.4. However, we continue to support Consul as a storage backend in production for our Vault Enterprise users. The decision on whether to migrate from Consul, or another existing storage backend, to the Integrated Storage backend is up to you and your operational requirements. In order to make this decision, it is important to understand the differences between using the Integrated Storage backend versus using an external storage backend. We recommend you start with the Preflight Checklist.

Once you familiarize yourself with all the information in the preflight checklist, should you choose to migrate from Consul to Integrated Storage, please review the Storage Migration Guide for Consul to Integrated Storage, that provides migration steps using the vault operator migrate CLI command.

»Next Steps

To learn more about Integrated Storage, please visit the Integrated Storage documentation, or these helpful learn guides:

• Vault with Integrated Storage Reference Architecture
• Preflight Checklist: Migrating to Integrated Storage
• Storage Migration Guide - Consul to Integrated Storage
• Vault HA Cluster with Integrated Storage
• Vault HA Cluster with Integrated Storage on AWS

Also, if you enjoy playing around with this type of stuff, maybe you’d be interested in working at HashiCorp too since we’re hiring!