5 steps to set up Vault for widespread adoption at your org

Once you’ve introduced HashiCorp Vault into your organization, the next step is to encourage developers and other teams to use it in their applications and services, while also helping them through their migration process. We’ve seen successful Vault adoption initiatives across small to very large companies (like Adobe), and we know which internal developer enablement practices work best.

This blog post outlines steps to drive the adoption of Vault within your internal developer community and sets the foundation for scaling secrets management over time. Based on your application ecosystem, the post will help you identify which secrets engines and authentication methods to enable, and it will help you evaluate whether a Vault Agent or client library (language-specific SDK) approach is the best way for your application to retrieve secrets.

»Discover existing applications and secrets

Ideally, before introducing Vault to development teams, you should first conduct a discovery exercise to investigate secret usage and application configuration in your environment. Knowing how services currently use secrets helps you plan for how to set up Vault and create education materials for development teams. This provides critical information on how developers can best adopt Vault.

For each application, record:

• Where it runs
• Its programming language
• Its framework

The runtime determines the Vault auth method you should use. The application’s framework and interaction with secrets determine whether you use an SDK or Vault Agent for the application to retrieve secrets from Vault.

»Make a list of the types of secrets

Make a list of the secret types — such as API tokens, certificates, and passwords — used by the application. The list doesn’t need to be exhaustive, but identifying the most common types of secrets will help you select the right secrets engines.

»Learn how the application retrieves and uses each secret

Many applications rely on environment variable injection, but separate processes inject the secrets into the application environment. For example, Amazon ECS tasks can set environment variables that reference secrets in AWS Secrets Manager. While not always possible, documenting how the secrets get injected helps you choose whether to use Vault Agent or an SDK for an application.

»Identify how the application handles secret rotation

Do you need to manually restart it if a password changes or will the application automatically refresh any connections using the secret? While you may discover that most applications use long-lived, static secrets, you might find others using auto-rotation managed by a cloud service provider. This information will help determine whether the application can handle the rotation of a secret with minimal downtime.

»Document any availability requirements for the application

Some applications require minimal downtime and are not allowed to restart because they are considered critical for business operations or have legacy architecture that does not support high availability. Other times, an application has sufficient replicas to serve requests and allow a rolling start of instances.

This table provides a template for discovery and assessment of applications (An editable version of the table exists on GitHub):

You do not have to survey every application in your organization, but getting a representative sample helps you identify application candidates for Vault migration. Identifying the most common runtime, framework, and secret types used by your applications will help your platform team determine the general settings for auth methods and secrets engines to enable in a typical Vault namespace.

»Decide developer access to Vault

After completing the discovery step, you can decide how you want developers to interact with Vault.

»Outline Vault namespaces

If you use HCP Vault or Vault Enterprise, you can isolate developer access to specific Vault namespaces. This pattern of delegated administrator management lets developers configure Vault themselves, with some constraints.

To design a namespace hierarchy, you can use our recommendations. Otherwise, you will need to organize and standardize the paths to your Vault secrets. Christopher McNabb's HashiTalks presentation offers a great example of how to establish Vault secret naming conventions using API paths.

»Use a custom interface to Vault or auth method alternative

If possible, avoid offering direct developer access to Vault. Use your own interface, such as a ticketing system, custom CLI, HCP Waypoint, or infrastructure as code to enable developers to update Vault. This lets you standardize and audit how developers store secrets in Vault while leveraging existing developer platform interfaces.

If you do not have a custom interface and need to enable developer access to Vault, choose an auth method that aligns with your existing identity provider (IdP) for defining development team groups (e.g. Okta auth method, LDAP auth method, whichever system you use).

»Use Vault policies and policy templating

Vault policies tied to groups should provide write access to specific paths and read access to secrets engines and auth methods. Use policy templating to scale your efforts to maintain and define policies.

»Offer flexible migration support

Developer access to Vault should support the initial migration of existing secrets, but your developer interaction approach may change over time. Some development teams will want to use different secrets engines after the the initial migration. Adding a workflow to grant temporary administrative access to specific development teams that need to use less-common secrets engines makes it easier for developers with outlier use cases to adopt Vault.

»Build an application landing zone

An application landing zone is an environment that is secure, repeatable, and highly customized to your organization. Similar to cloud landing zones (AWS, Azure, Google Cloud), an application landing zone for Vault contains the minimum required features for your teams and applications to get started with Vault. This includes items such as a namespace, the key-value secrets engine, templated policies, machine authentication methods, and human authentication methods. Use the data collected during the discovery step to identify the initial set of secrets engines and authentication methods to enable by default in the application landing zone.

»Use a landing zone accelerator

A landing zone accelerator lets platform teams quickly create application landing zones. Accelerators can be built as HashiCorp Terraform modules or with scripts. The accelerator interface could be managed through HCP Waypoint or your own interface, such as a ticketing system.

Depending on your organization’s strategy, the accelerator can follow a self-service approach, allowing end users to onboard themselves to Vault. The application landing zone accelerator should handle the initial configuration and setup of Vault authentication and mounts. Additionally, the accelerator should configure third-party platforms like AWS, Azure, Google Cloud, or Kubernetes.

»Add new golden patterns to landing zones

An application landing zone provides a standardized framework for adopting Vault at scale and should evolve with your organization's needs over time. For example, you can observe the usage of early Vault adopters and build new golden patterns into the landing zone to better onboard the rest of the organization.

»Migrate secrets, update application

Most applications need to have some refactoring before using Vault because they do not currently support live reloading of new secrets. As a result, applications or their deployment manifests will need additional code or configuration to recognize new secrets from Vault. We recommend that you start with a non-critical application to test your Vault integration process and iron out any issues.

Choose an application that:

• Depends on one (or very few) static secrets
• Uses a framework or runtime with Vault integration support (see list)
• Can handle some downtime

The first step involves migrating your secrets for that application into Vault.

»Migrating secrets

Above we suggested that you pick an application with very few static secrets to migrate. An application with fewer static secrets means you learn the process of transferring the secrets into Vault and minimize the blast radius if the application refactor does not succeed on the first try. Eventually, you need to duplicate the migration process for the application’s secrets for both non-production and production Vault clusters. Note that you likely will need to copy only the latest version of the secret into Vault, not the previous versions.

If the application references secrets that are automatically rotated by a cloud service provider (such as a database username and password), copy the secret into Vault and manage it as a static role (if possible) or key-value secret. Using a static role with an existing username and password ensures that you can rotate the password with Vault at a future date. If the database secrets engine does not support static roles, use key-value to store the username and password. Dynamic secrets have leases, which an application must handle differently. Over time, you may consider replatforming static secrets to dynamic ones.

»Updating the application

Ideally, you want to minimize the overall refactoring effort. However, if you maintain consistent development practices across your application teams, such as the use of standardized frameworks like Spring Boot, or retrieve all application configurations from a configuration server, you may be better served taking a Vault-aware approach to updating your applications for Vault. This is the more refactor-heavy option in which you refactor your application to interface directly with Vault.

If these strict development practices are not part of your application delivery process, we generally recommend taking a Vault-agnostic approach by deploying Vault Agent or one of the Kubernetes integrations (see comparison) with your application. These utilities handle Vault authentication and secret reloading into environment variables or a file, and they require only minor code additions.

»About Vault-aware direct integration

For direct integration with Vault, you will need to use a Vault client library (also known as SDKs) to request secrets from Vault and handle reloading them. As mentioned before, this increases the overall refactoring effort for the application. One scenario in which you need to use a client library and implement a Vault-aware integration includes encrypting application data using the Vault transit secrets engine.

Another Vault-aware scenario involves an application that interfaces with an existing secrets manager. In this case, you will need to refactor your code to change the secrets manager to Vault as part of the application properties. You’ll want to look for tutorials on reloading secrets in your specific application framework. For example, if you use Spring Boot, this tutorial on reloading secrets in Spring applications would help you identify what code requires refactoring.

Using a client library to connect the application to Vault requires additional guidance from your security team, as they will need to verify software dependencies. Some, but not all, client libraries handle application reloads for new credentials or Vault token refreshes. Application teams will have to implement this logic themselves.

If you have to write your own code to reload the application, you can deploy Vault proxy to handle the initial authentication, which eliminates the need for Vault authentication logic in your application code. Vault proxy enables the application to connect to the API proxy to retrieve secrets from Vault.

»About Vault Agent

Vault Agent runs as a separate process and handles the authentication, retrieval, and injection of secrets from Vault. Vault Agent can also restart the application with new secrets using the exec directive command, either by issuing a termination signal or API request to the application. It's often the preferred Vault integration approach when you don’t require a Vault-aware approach and you don’t use an orchestrator like Kubernetes or HashiCorp Nomad. If you do use one of those orchestrators, read their sections below.

»For Kubernetes users

If you are on Kubernetes, you may prefer to use Vault Secrets Operator as a more direct integration between Vault and Kubernetes. If your test application for Vault integration runs on Kubernetes, try deploying the Vault Secrets Operator onto the Kubernetes cluster and defining custom resources to synchronize secrets from Vault into Kubernetes Secrets. You don’t need to refactor the application deployment manifest if it references the Kubernetes secret and injects secrets into environment variables. This approach requires a minimal level of effort.

»For HashiCorp Nomad users

If your application runs on HashiCorp Nomad, configure the Nomad-Vault integration to securely inject secrets into your application.

If you do not have Nomad in your environment, or you run on virtual machines, consider using Vault Agent or a proxy for applications that do not run on Kubernetes.

»Another option: Secret sync

If you encounter significant obstacles that prevent the application from using Vault Agent or a client library, consider using secrets sync. This Vault Enterprise and HCP Vault Secrets feature allows teams to define their secrets in Vault and synchronize them to a target API, such as a cloud service provider’s secrets manager or CI/CD framework (e.g. AWS Secrets Manager, GitHub Actions, etc.). With secrets sync, applications can use existing interfaces without refactoring, while you secure and track secrets in Vault.

»Take note of issues encountered

As you copy the secrets to Vault and refactor the application or its deployment manifests, you will incur downtime when restarting the application to retrieve the new secret. If an application does not reload after you’ve rotated the secrets, you may also have to add application logic to handle a reload gracefully. Applications using Vault Agent should reload once configuration properties change or include handling for graceful shutdowns when getting a termination signal. Document the steps for refactoring and any gaps you identify in provisioning the application landing zone, configuring secrets engines, and refactoring the application.

»Organize internal developer advocacy

Properly supporting developer adoption of Vault requires documentation and education on how to use Vault properly in your organization. After the initial refactor, create documentation and tutorials for any patterns you employed specific to your applications. Our tutorials and documentation on application integration with Vault are a great place to start, but you should also create custom materials for your organization. Our videos on reloading static and dynamic secrets in Spring applications are great examples of developer-enablement content.

The documentation you provide should include:

• Onboarding your development team to use Vault
  • Description of the default developer Vault policy, such as read-only or write for static secrets
  • Parameters to access a given Vault namespace or secrets path
  • “Break-glass” instructions if developers must access Vault directly
• How to store static secrets
• How to enable and configure dynamic secrets engines
• Create code examples for static and dynamic secrets using client libraries, Vault Agent, Vault proxy, or another controller

Many organizations create workshops, brown bag sessions, coding challenges, and hackathons to support the initial application refactor and expand adoption. We also recommend regular office hours to help development teams use Vault. Offering regular onboarding support to development teams helps encourage Vault adoption, identify new usage trends (such as popular secrets engines), and correct unsustainable patterns. Consistent feedback from your developer community helps you evolve secrets management as a platform product offering and provide standard patterns to manage secrets across applications.

»Grow Vault usage

After you onboard an initial set of applications to use Vault, think about whether you need to create any new documentation or features for the next set of applications. For example, you may want to expand the default set of secrets engines you enable for developers, such as SSH or PKI secrets engines. If application developers need the ability to encrypt and decrypt payloads, you may also want to enable the transit secrets engine. Note that the transit secrets engine requires applications to use a client library to encrypt and decrypt application data.

Your initial secrets management landing zone will likely change over time as more developers onboard. As more developers become familiar with storing and retrieving secrets from Vault, consider refactoring application CI/CD pipelines to retrieve secrets from Vault.

For static secrets that have an expiration or rotation interval, such as certificates or database credentials, consider introducing dynamic secrets engines. Dynamic secrets work differently than rotated or static secrets, as they involve a new set of usernames or passwords each time.

The Vault-agnostic approach tracks the expiration time of the old secret. The Vault-aware approach requires the application to track the lifecycle of secrets and request a new secret before the old one expires. As a result, dynamic secrets engines require additional application refactoring to track the expiration of secrets.

Besides working with application teams, be sure to include security stakeholders when discussing the time-to-live for dynamic secrets and specific configuration options for the secrets engine. For example, the security team responsible for certificate management should provide input on which intermediate certificate authority to use for certificates. In the case of dynamic database credentials, the database team should participate in deciding the database policy attached to the secret (such as allowing read-only access to the database) and tuning the database to create and delete users.

»Learn more

For more information on Vault, review our documentation on secrets engines and authentication methods. As you set up Vault, review our guides on recommended patterns. When refactoring an application, reference our tutorials on integrating applications using Vault Agent or SDKs.