Vault benchmark testing tool

Load testing is an important part of releasing a reliable API or application. It gives organizations the confidence that the infrastructure, applications, and workloads will work well under a defined load. This testing is also important to ensure that self-managed HashiCorp Vault clusters can handle a large number of concurrent requests — sometimes thousands per second — in a real-world scenario. Because if Vault goes down, secure access to secrets and encryption as a service go down.

This post discusses how the Vault benchmark tool our engineers have built can give operators confidence in the reliability of their secrets management lifecycle.

Vault benchmark is designed to test the performance of Vault authentication methods and secrets engines. Its load-testing capabilities are powered by the HTTP load testing utility Vegeta.

»Vault benchmark benefits

The benefits of benchmarking your Vault workloads include:

• Increased uptime due to the ability to spot future problems.
• Breaking down operational silos by monitoring across system components, enabling the teams responsible for those components to come together to fix issues.
• Faster response to problems by collecting real-time data.
• Increased innovation when IT teams have more time to work on mission-critical projects rather than fixing application problems.

»Vault benchmark prerequisites

Prerequisites for using Vault benchmark include:

• Familiarity with using the command line (installing and executing CLI apps)
• The application/API deployed on a server (dev/staging) for testing. You may use Vault benchmark for local tests but they might not give an accurate picture of how the server will behave under a real-world load.
• Experience using Vault.

»Using Vault benchmark

To use Vault benchmark, run the vault-benchmark binary along with a benchmark configuration file. Use the file to configure any resources on the Vault instance that are required to perform tests. Before running the binary, set up any infrastructure dependencies, such as a database.

Depending on the configuration, Vault benchmark may put a great deal of stress on the Vault cluster and the underlying infrastructure during testing. Vault benchmark is intended to be run against non-production Vault clusters that are isolated from production systems or any other systems that might negatively impact the end-user experience.

Through load testing, engineering teams may discover aspects of their architecture that are performing well, as well as opportunities for improvement. We recommend using Vault’s production hardening guidelines and reference architecture in addition to load testing to improve and tune overall performance.

You can download the Vault benchmark release binary from our release page. Documentation for Vault benchmark, which includes usage examples and test configurations, can be found in the project’s GitHub repository docs folder.

To learn how Indeed manages the reliability of its Vault clusters and uses Vault benchmark, watch the HashiConf 2023 talk: All the 9s: Keeping Vault resilient and reliable

“[Vault benchmark] makes codifying your regular traffic and benchmarking clusters really easy. Just like you define your infrastructure as code, you define your traffic patterns and stress tests as code.

The tool already supports a comprehensive set of off-backends and secret engines, so you can easily map your standard client interactions and reproduce them in lower environments. This is an invaluable tool to understand how your current configuration and, more importantly, how any future configuration changes could impact cluster performance.” — Mark Billow, SRE, Indeed

»Self-managed and HashiCorp-managed Vault

For more information about best practices for self-managed Vault Community and Vault Enterprise, visit HashiCorp Developer and check out the full Vault knowledge base. If you’re interested in outsourcing Vault operations and reliability engineering to HashiCorp, try HCP Vault or HCP Vault Secrets, our cloud-managed versions of HashiCorp Vault.