HashiCorp Vault: Zero to Hero

Transcript

Hi, everybody. Welcome to HashiConf Digital 2020. And thank you for joining my session, titled "HashiCorp Vault Zero to Hero.”

My name is Brian Krausen, I'm a principal consultant, and I've been working with Vault for probably 3.5 to 4 years now.

I've had the opportunity to work with a lot of Fortune 1,000 companies, designing and implementing Vault.

I'm super excited to be able to bring you this presentation and bring you my experience with working with Vault with other customers.

Let's start off with Vault. Vault can be somewhat complex when you first start learning it. When you start looking at Vault, you worry about things like authentication methods, secrets engines, how to unseal Vault, all those kinds of things.

And honestly, they can be a little daunting when you're trying to deploy in your organization. However, it's not that bad once you finally understand all the different components and how they're all put together.

The goal of this presentation is to walk you from going from Vault, zero knowledge, all the way to becoming the hero in your organization by deploying Vault and getting them up and running.

Operationalizing Vault

The first thing you want to do is to build the right team for your organization. I've seen customers where maybe it's 1 person, 2 people that get up to speed on Vault, and they're the ones who run Vault and manage Vault and start integrating with their applications.

But oftentimes that 1- or 2-person team isn't enough. We want to make sure we're involving different skillsets when we're building our Vault team.

Embrace other teams within the organization. This could be the first time that you interact with these teams. Become friends with security. They're not always the bad guys.

When we're building the right team, we want to think about things like the operational team. The operations team is going to manage Vault on a day-to-day basis for you.

We need to involve the infrastructure team, because the infrastructure team may own the Vault infrastructure, the VMs if you're on-premises, or things like instances if you're running in the cloud environment.

They need to fully understand the Vault environment as well so they can help support it if something goes wrong.

You need to involve the cloud team, assuming you're running on a public cloud of some sort.

And then the security team.

A lot of times when I'm working with customers, I see the automation team and the cloud team are the ones who are trying to bring Vault to the organization. And oftentimes security is not involved.

I highly discourage that. because security is going to want to use Vault as well. And they're going to use Vault to improve the security posture across the organization, and they're also going to be the ones that help champion Vault to the organization.

Your project throughout the business is going to help a lot of different customers. So from different developing teams to just IT to even things like HR, whoever else wants to use Vault.

Training the Team

Once we build the right team, we want to make sure we train that right team.

I've worked with customers who really love the idea of the features and functionalities that Vault brought to their organization, but they didn't train the team as they should have.

One customer really relied on me for doing everything within Vault. By the time I left that customer, they hadn't had anybody who went out and did training. They didn't fully understand the Vault architecture, how to work with Vault, things like that. That's just a bad situation.

You can use a HashiCorp partner to deliver this type of training, you can go through official HashiCorp training, or there's a lot of community-based content out there. Regardless of which one you do, you want to make sure that you accurately train your team so they understand what's going on within the Vault environment.

Automate Upfront

The last thing I wanted to talk about operationalizing Vault is you want to make sure you automate everything upfront. Use things like Terraform or other infrastructure-as-code tools to deploy your Vault and Consul nodes, assuming you're using Consul, and also configure Vault.

That not only allows you to get Vault up and running quickly, but it also allows you to deploy other clusters of Vault if you start extending Vault to other datacenters. Or if you happen to have an outage and need to quickly deploy Vault to another region or another datacenter, you have that code that you're able to automate and bring Vault up pretty quickly.

The second thing is, don't reinvent the wheel. There's tons of content out there. Search GitHub and look at HashiCorp's website, especially the HashiCorp Learn platform, things like that. Don't start from scratch.

There's a lot of community content out there that you can use and then build upon that content to automate Vault the way that you need it to be automated.

The other thing is: engage a HashiCorp partner if you're unsure about how to automate the deployment of Vault and get it configured. You don't want to spend a lot of time working on automation when you could quickly hire somebody for cheap to come in and lay down that configuration. They can bring that experience that they have working with other customers to your organization.

Now you have a fully functional, vetted Vault deployment upfront versus doing it how you know how to do it and then slowly having to improve the environment.

The last option here for automation is working with the HashiCorp implementation services team. This is not a sales pitch for the team; I don't work for HashiCorp. But it's a great team that you can use to run a quick start on Vault and get it up and running in your environment very quickly.

Integrating Vault with Your Applications

The first thing that I typically will talk to customers about is trying to centralize that identity for applications around Vault.

We can talk to our different development teams and say, "Hey, if you're ever looking for credentials, come to me, come to Vault. Vault can manage credentials across all these different platforms, and then you don't have to worry about integrating with these individual platforms. So you don't have to worry about understanding Azure deeply or AWS or Google or Kubernetes or any of those kinds of things. All you have to understand as a developer is how to interact with Vault, and you let the Vault team manage everything else."

It makes it a lot easier for the developer.

Migrate Your Secrets

Once you integrate things, you want to start trying to migrate some of those static secrets within your organization that you have laying around.

You don't have to pretend they don't exist; I know they exist in every organization. I've been an administrator, and I've had those service accounts that have a password that has been there for 3 years and it's being used for the application to connect to the database.

Think of those types of credentials that are within your organization, and those are the things that you want to try to migrate into Vault, because they tend to be the easiest credentials to migrate.

Think about things like your regular Active Directory accounts, database service accounts, security scanning. If you have a vulnerability scanner going out to scan everything on a nightly basis or a weekly basis, those types of credentials are generally privileged credentials that you don't to want last very long or fall into the wrong hands.

You want to definitely take care of those types of credentials. You want to take care of public cloud credentials. And that's not only saying I need to store those static credentials, but we also want to be able to create those dynamically on the fly.

Finally, once you get all those static credentials in Vault, we want to update all those to dynamic credentials if possible.

Once you get those static credentials in there and you have applications integrating with Vault, meaning they go to Vault and they can access a certain path for a secrets engine in Vault—for static secrets, it's probably going to be the KV store—once they can successfully access the KV store and get those credentials, you can start migrating some of those applications over to dynamic credentials.

For example, if you have a database account that's being used to access MySQL account somewhere, either on-premises or things like Amazon AWS, instead of having that credential last forever, 24/7, and only update that maybe once a year, you can enable the database secrets engine and integrate that database secrets engine with that MySQL backend.

Now you go to the application team and say, "Hey, instead of going to KV database to get your static credential, go to databases in the role name. What happens is Vault will generate that credential dynamically for you with a very short TTL. And every time that application comes to Vault to get a new credential, then Vault will respond with a new credential that's been rotated."

Think of those credentials that you have to rotate every single year to meet PCI requirements or things like that. If you're integrating that with Vault, you're rotating that every 15 minutes or every hour, instead of waiting that full 365 days.

Definitely something that you want to look at is migrating static credentials, and then moving those static credentials over to dynamic credentials if possible.

Once you get all those credentials in Vault, you can start updating all your CI/CD tools to take advantage of Vault. Instead of storing secrets in each individual CI/CD platform like Jenkins Secrets or Ansible or things like that, consolidate into Vault and then have your CI/CD platform reach into Vault and get those secrets.

You can do the same thing with your CI/CD platform with Vault that we were talking about in regards to static secrets. Move those static secrets over, have a CI/CD application come grab them from Vault, and eventually migrate those over to dynamic credentials.

Now, every time the CI/CD job runs, it's generating those dynamic credentials. Maybe they're automatically revoked at the end or they have a very short TTL.

With things like Jenkins, you can install a plugin to integrate directly with Vault. Or you can simply use things like the CLI or API to retrieve those secrets.

Finally, and this is more of a long-term play, once you've got your easy static credentials in there and you've migrated your CI/CD applications to use Vault, let's start thinking about, How do we refactor our applications directly to integrate with Vault?

I get that a total refactor or rewrite of an application is probably a long project, maybe a year or several-year project. But we can start using things like the Vault agent or envconsul or Consul Template to have those tools integrate with Vault.

Then the application only has to integrate with those tools, or it doesn't even have to integrate with them all. It doesn't even have to know it exists. Now your local application can rely on something like Consul Template to go out to Vault and grab a secret and then automatically update a configuration file with a new secret.

Or envconsul can go out, grab a secret from Vault, and then it can drop the environment variables in regards to the new credentials. You can start immediately integrating applications with Vault, but as you work on a much larger project of refactoring those applications to take advantage of the services that Vault's providing.

Be a Vault Hero

Once you start integrating those applications and having those quick wins and things like that throughout the business, you want to make sure that you're showing value to the business. And this is really the hero part of the conversation.

You want to make sure that the business understands what you're doing with Vault and how it's improving the organizational security posture for your applications.

The first thing you want to do is talk to security, talk to management, things like that, and address some of these organizational challenges that you may have had, things like rotating passwords.

You can also offer up things like data encryption. So instead of having to buy a secondary tool and managing that tool and pay for the cost of that tool and all that kind of stuff, let Vault encrypt that clear-text data for you, and then you can store that on whatever backend you want.

That allows you to be less reliant on that backend that only does the encryption that you require. And then you can have Vault encrypt that data and you can place that wherever you want.

Vault can do things like tokenization, which may be valuable to financial or retail organizations. PKI certificates. Secrets consolidation is a big one that I generally see with a lot of customers. You might have somebody using one password for everything or using the previous password or storing passwords in an Excel file.

Have all those people consolidate all their secrets within Vault. And that also allows you to gain access to those credentials in case somebody goes on PTO or they're unavailable during a meeting, things like that. They're readily available for everyone that has access to Vault.

Finally, you can use Vault to limit privileged access for people. You can have Vault create certain policies in the cloud as people are generating those dynamic credentials. So if somebody only needs access to read an S3 bucket, you give them the credentials, and the policy attached to those credentials only allows them to read the bucket. It doesn't allow them to spin up EC2 or things like that.

Once you start addressing a lot of these challenges, the final thing you want to do is to definitely communicate this to the organization. Start meeting with stakeholders of the applications to show them the things that you can do and the things that you have done already to improve the security posture of their application and the organization as a whole.

One thing I like to do is market the benefits of Vault to the organization as a whole.

As you start building rapport with different developer teams, they're going to become advocates for Vault, and they're going to start telling all the other developers. You may have an influx of requests into Vault once you start successfully integrating applications with Vault.

But you want to make sure you're marketing the benefits to Vault so everybody's aware that Vault is in the organization and it can do all these different things for your application, to make it easier for development teams or the organization as a whole to make sure that everything is secure.

The last thing is, make sure you demonstrate those integrations. As you have quick wins within the organization, or if you have specific integrations that you want to show off before they're actually implemented, don't be afraid to show those off.

Even if you don't have a development cluster or anything like that, you can easily run Vault locally on your laptop and configure those integrations. I do it all the time with customers or demonstrations and things like that. Communication to me is really the way that you're going to become more the Vault hero in showing the business what you've done and what you can do with Vault.

A Quick Recap

First, we want to operationalize Vault, get it up and running as fast as possible. Don't waste a lot of time with that. Engage a partner, engage HashiCorp directly, use community resources.

You want to integrate that with your applications quickly. So move those static secrets in, and change them to dynamic.

Then start thinking about the long-term effects of integrating applications into Vault.

And once you get success, show that value to the organization.

I hope this quick presentation gives you some ammo as you start to deploy Vault within your organization and try to gain a lot of value and show the organization that Vault is going to be a very valuable tool for you.

My good friend Rob Barnes and I recorded a HashiCast about this topic. The idea is that I only had 25 minutes to talk about some of these things at HashiConf Digital, and we wanted to be able to expand on that. So look for that coming soon after HashiConf Digital.

Dan McTeer and I have a book coming out about running HashiCorp Vault in production. Dan ran and deployed Vault at Adobe and he now works for HashiCorp.

I hope you enjoyed the presentation, and thanks for watching.