How to Adopt a Producer-Consumer Model for HashiCorp Vault

Setting up a central service for HashiCorp Vault can be challenging due to the shared ownership of the secrets and data between the functional team that is using the secret (whether it’s operations, infrastructure, data, engineering, etc.) and the team that is operating Vault and securing it. Requiring that all of these functional teams acquire and maintain expertise in cryptography, secure secret use and management, and machine identity can slow down progress towards important strategic initiatives like building new applications. That’s why many organizations opt for a producer-consumer pattern, which shifts those specializations to the team operating Vault.

• The “Producer” team owns the Vault service itself and the use-case-based requirements overall.
• The “Consumer” teams own the services that utilize or consume the Vault service.

Having worked with hundreds of our customers who have adopted this producer-consumer model, we have identified the most common challenges organizations run into. In this blog, I will identify these challenges and share the best practices for solving them at scale. I will also link to several tools for implementing these best practices, including a use case matrix, intake template, and questionnaire.

»The Challenges of a Vault Producer-Consumer Model

Each team has its own set of objectives, metrics, and challenges when dealing with secrets management. At times, their metrics and objectives can be at odds, which can result in slow progress, poor security practices, and a failure to create a scalable, efficient Vault workflow that works for everyone.

To make sure this result doesn’t happen, it is important to understand and empathize with the various consumer-side stakeholders’ perspectives:

• Security Teams:
  • Goal: Decrease risk to the company and its customers.
  • Responsibilities: Protecting sensitive data, ensuring access to sensitive data is restricted, assessing the risk level across the organization.
  • Key challenge: Security is not the primary custodian of sensitive data and therefore must rely on cross-functional influence and relationships to achieve its goals.
• Engineering Teams:
  • Goal: Deliver new services or data that creates a competitive advantage for the company as fast as possible.
  • Responsibilities: Create new digital services and capabilities for their company.
  • Key Challenge: High reliance on cross-functional stakeholders such as infrastructure and cloud teams, security teams, networking teams, and financial teams to be able to achieve its goals. Burdens of collaboration can decrease agility and slow time-to-market.
• Operations Teams:
  • Goal: Ensure consistency and reliability in achieving company goals.
  • Responsibilities: Maintaining system availability/uptime, providing financial predictability, and delivering required resources to cross-functional stakeholders.
  • Key Challenge: Requirements for operating a business and maintaining a competitive advantage can be at odds with the changing landscape of new technologies. Likely not specialized at the level of cross-functional stakeholders like security or engineering.

Overall, the producers must view themselves as providing a service on behalf of the consumers. Consumers hold the keys to their respective use cases and can choose whether to participate or not. It is incumbent on the producer team to make it easier to accept and adopt the central service.

»Best Practices

Starting off on the right track requires a partnership between all of the stakeholders. In many organizations, top-down mandates can struggle to find success without buy-in at all levels. Greater success can be achieved by placing the needs of all stakeholders at the forefront.

»1. Vault as a First-Class Citizen

When Vault is an essential part of all planning, Vault becomes a critical component of your zero trust strategy or security initiative. It’s incredibly difficult to achieve a robust producer-consumer model otherwise. As a critical central service, Vault requires adequate engineering resources to ensure the success of the deployment, operations, and ongoing adoption. Success requires continued focus on training and driving adoption across your organization — the job is not complete when the first use case is implemented.

»2. Understanding Needs of Producers and Consumers

It may be that there are too many teams for it to be feasible to meet with each. If this is the case, pick a representative sample of teams to work with on requirements gathering. This can include the most stringent use cases in terms of security or compliance, which may naturally allow you to build the service out in a way that represents the largest number of teams or services possible. If you choose a sample of representative teams to collect requirements for, this will map to the stakeholders section of the service-level agreement (SLA).

It can be extremely valuable to have a complete understanding of what the consumers need and then define what the producer team needs to build into Vault. In order to serve many internal constituents, documenting every use case and the applicable needs is required.

Using the use case matrix, intake template, and questionnaire linked below in this article can help you accelerate progress while the sample questions and guidance from this post will accelerate adoption.

»Set Up Questionnaire

When considering setting up the producer-consumer model, there can be a lot to think about - resiliency requirements, use cases, the geography of workloads, rotation requirements, and approved authentication methods. Here is a link to a questionnaire for defining operating principles for the Vault central service producers and consumers. This will help teams define the service and document requirements so that the Vault service is set up correctly, ensure that knowledge is not lost over time, and help with consumer onboarding.

It includes questions such as:

Sample Consumer Question:

• Which services, tools, and users will be interacting with the service and how will these interact with Vault?

Sample Producer Questions:

• What is the password lifecycle/rotation requirement?
• What are the prescribed authentication methods?

Here is a link to the full operating principles questionnaire.

»Consumer Intake Form

As new consumer teams are ready to be onboarded to the central service, baseline information must be collected. Oftentimes, consumer teams are not experts in secrets management, encryption, or machine identity so they may not know what information to share with the producer team. Some of the important categories of information that must be collected are who the people are who own the use case, what functional requirements they may have, performance requirements, or any unique circumstances of their use case.

Here is a sample intake form that can be used by teams that are currently using Vault or new teams that need to start being served by the Vault Central Service. This form can be edited to take any unique requirements into account.

Here is a link to an intake form template.

»Program Management: Use Case Matrix

The Use Case Matrix is a project management framework that will enable you to split the requirements of the producer team and the various consumer teams that will subscribe. By breaking out the requirements in this way, it is easier for the producer team to manage their existing consumers and onboard new consumer use cases. Over time, this will serve as program documentation as business needs and personnel change.

Here is a link to a use case matrix template you can use to document your program.

»3. Proactive Engagement with Consumers

The producer team must reach out to consumer services owners across their organization to understand their needs for identity management, encryption, and secrets utilization. Three best practices for engaging with current and future consumers of Vault are:

1. Each user/team consuming Vault should fill out a Vault service-level requirements (SLR) template — no exceptions. This document should be maintained in a central wiki or some other repository where it can be referenced by the consumer team, the producer team, or anyone else that may need access to the document. This can serve many purposes, such as solving disputes around agreed service levels, providing context to internal auditing teams on how a process consumes secrets, or as a reference for the service team as they build out and manage the service.
2. The producer team should keep meeting notes and record them in the SLR document for any team that they meet with face-to-face. This helps to ensure the documentation and defined objectives align. Again, a meeting with each team is not always necessary, but notes should definitely be retained for each team you do meet with.
3. Follow up with each team every six months to a year. This can likely be automated and come in the form of an email containing the link of the current SLR document and inquiring as to whether there are any planned changes that may affect the consumption of secrets.

»Final Advice

Often the producer team that owns and manages Vault is focused on security and operational objectives. One such objective is getting as close as possible to 100% of all secrets being dynamically created, served, timed out, and destroyed on demand. Another objective should be to give consumers a customer-level experience since their adoption of your process will be key to reaching that 100% level of secure secrets management. Producers need to give consumer teams the easiest possible workflow so they can be focused on developing new application functionality. Producers also need to provide a reliable, highly-available deployment of Vault so that consumers can trust that their secrets, authentication, and encryption services will be available when they need them.

If you would like to learn more about building these programs at scale or would like our help, please reach out to our sales team or your HashiCorp representative.

»Resources List

• Operating principles questionnaire
• Intake form template
• Use case matrix template