Manage SSH with HashiCorp Vault

Vault is part of many enterprises' full featured secrets management solution, and provides a bridge between many layers of the stack and the information security teams. The pluggable nature of Vault's secrets engine and authentication backends make it an attractive offering for enterprises. The SSH one-time-password capability and the SSH key signing capabilities of Vault are two use cases that have been broadly adopted by many enterprises—this talk will discuss how that is accomplished. It will not necessarily go into the architecture of Vault, but more about building the policies, roles, and credentials associated with these two use cases. Additionally, a demonstration will be shown to provide context around how Vault and SSH can be integrated.

Transcript

Hi, thanks for coming. My name is Erik Rygg, I'm an enterprise architect at HashiCorp, and I'm going to talk to you a little bit about SSH. Specifically, managing it with Vault.

What's the problem?

Let's take a look at what the problem set is, what we're trying to solve with using Vault to manage your SSH keys.

The issue here is that not a lot of enterprises are in a state where they can manage immutable infrastructure. We're still using VMs, we're still trying to realize the dream of a utopian containerized and immutable world. Which is great, that's a great future state to be in, but the problem is that we still have to manage our SSH keys.

They need access to instances and running services, and viewing logs, troubleshooting issues. We're still using VMs, physical hosts, mainframes, things like that. It comes down to, you have to manage your infrastructure still using SSH.

What's the problem with using SSH? There are a lot of issues with key sprawl. A lot of people have used GitHub to store PEMs. You might have PEM files sitting in a home directory. I've had an issue where, in the middle of the night, I've had to call my co-worker to say, "Hey, your PEM file is in your home directory, and I need to access this instance."

That's completely crazy, but it's reality. PEM keys are all over the place. We're having to manage complex systems for propagating SSH keys. Whether it's Chef, or I've used active directory in the past to push out SSH keys.

The problem with using Chef or Puppet, or something like that, is oftentimes we wonder, "Did that SSH end up on the instance that I need it to? Did it clean up the instances? When my buddy left the company, does he have it in his home directory? Can he still access my systems?" We're having a lot of issues with managing those keys and they're all over the place. Not to mention group accounts, service accounts, they are often looked at as a no-no, because if I give my key to everyone, then everyone has access to that group account.

Additionally, we have accounting. There really is no accountability with respect to managing your SSH keys. You can't manage the lifecycle of access of your systems using SSH, so Vault provides that as well. What happens when users leave the company or they leave the team? Do you want them to have access to the instances that they once did? Because, they'll continue to have that as long as they have that PEM key.

Vault, the Swiss Army knife of security tools

Vault has a lot of capabilities. It's the Swiss Army knife of security tools, and SSH happens to be one of them. The nice thing about using Vault for SSH is that you can create unique credentials on demand.

There are two main features of using Vault and SSH. You can use one-time passwords. I'll show you how you can configure your instances to use a one-time password with Vault. Also, you can use it as a certificate authority for managing your SSH keys. Vault can actually sign your SSH key and you use that to access your instances. Assigning those keys, you've given authorization for that user to access your instances that you've created policies for.

Additionally, the SSH credentials can have a TTL associated with them. This allows you to auto-revoke those credentials as well. Vault itself, you can set up auditing so that when you access Vault to sign your SSH keys, you can see what's going on in your system. Who's accessing it, what policies they're using, things like that. It really helps with satisfying a lot of the security requirements that you have with enterprise systems.

The bottom line is, Vault and SSH allow you to build a workflow that efficiently and securely manages access to your systems.

The workflow for managing access using SSH one-time passwords

Let's take a look at what the workflow would be for managing access using SSH one-time passwords. Let's start out with three objects. We've got a user, Vault, and then your SSH clients themselves.

These SSH clients have a little helper utility on them that is configured using PAM: pluggable authentication modules. I'll show you how that's configured.

The workflow here is, a user, once he or she is authenticated to Vault, can make a request to Vault to say, "I need a one-time password, for my instance." Vault will say, "Yes, this person has access to be able to get a one-time password back," and will respond with that one-time password.

At that point, the user will then try to log in as the user that they gave access to the SSH client, and pass in that one-time password. At that point, the Vault SSH helper takes over and has an active connection to Vault to say, yes, this user, this password, and this policy can then be authenticated. The user then is allowed in.

The other option is using Vault as a certificate authority for signing your SSH keys. We have the same three objects. A user, Vault, and the SSH client. The user has an SSH key. He'll send that key to Vault to then get signed. Once that signing happens, it'll respond back with a new signed key, after validating that that user has access to be able to SSH to the instances they're making a request for.

The user then takes that key and can log in to the client. The client is configured with a certificate authority that is created from Vault. When that SSH occurs, there's not an active connection from the clients themselves to Vault. You've already used a trusted system to push that certificate authority to the SSH client itself.

That validation occurs locally. There's not that active connection, so that can be a benefit for accessing Vault. You could use one-time passwords if you're okay with your instances and your clients being able to access the Vault instance itself. Or you can completely network segregate them, and push the certificate authority to that instance and be able to SSH. A give and take, if you will. The main difference between one-time password and CA is that you can delegate how the delegation works with respect to access.

Let's take a look at what the scenario in the demo will look like. Halloween's coming up, so we're going to create an Undead Vault. In this Undead Vault, we've got vampires, and Bob's a vampire. He's a user, he's going to use one-time passwords to be able to access his instances.

We also have Suzy, who's a zombie. She's going to be able to access her instances using a certificate authority. One-time password again, for vampires only. We've set up the policies within Vault to say, we can only access the vampires-only instances as a vampire.

If I'm a zombie, I'm going to access, using the certificate authority, zombies only. Let's take a look at what this looks like in real life. We're going to log in first. A main tenet of Vault is authentication and then authorization. You have to authenticate the Vault first in order to get the policies that are associated with you.

Bob is allowed to log in to Vault using his username. He's going to get a token back. Once that happens, in Vault we have policies associated with that user. I'll show you that here in a sec. Right now I'm only using a simple authentication mechanism, userpass, and I put in the wrong password.

Let's take a look at the UI while we're trying to figure that out. The vampires are set up as one-time password users and the zombies are set up as CA users. Let's take a look at the details in the UI here. The role that I set up for vampires is that they have a key type of one-time password. The default user is Bob, and the port 22, and allowed users are Bob. Obviously, you could put different users in here. You could also use group users.

I know group users has been historically relatively not great to use, but here we can use that because now we're assigning authentication from Vault to specific users. The userpass that I was showing you, you can set up specific users to use group users for specific IP addresses. CIDR blocks.

Here, I'm setting it for everything, but you can break this down into localized groups. If you have development machines that you want certain users to access versus others, you could set that here as well. For the zombies, let's take a look at their policy. It looks a little bit different, because we're using a certificate authority. The key type is certificate authority.

The great thing about certificate authorities is, when you make that request for your signed key, you get a TTL that's associated with that. Here I'm setting it for 30 seconds. Once I've authenticated and I've made my request for assigning the certificate authority, I get that back, and it's only allowed to SSH to the instance for 30 seconds. That's it.

Let's take a look, make sure that we've got a Vault token. We do, that's great. We're going to write to Vault. I've authenticated to Vault as Bob, the vampire, and I'm going to make a request to Vault to get my one-time password back. But I need to give it an IP address, because like I showed you in that policy, we've got a CIDR block.

It is an open CIDR block, so if this was set up to be a specific CIDR block, then I could make that request and say, “Well, this IP isn't within that CIDR range." Then I would fail to write the credentials here. But since we're opened up, we're going to be able to do this. I did this in Terraform Enterprise, so I'm gonna go ahead and grab my IP address from the output here. If I had the new capability of doing remote plan and apply, I could do this on the command line here.

I've got my key now. Let's take a look at this here. I've made the request for this IP, one-time password using Bob, and I've got my key here. This is my one-time password, the key. I'm going to SSH to that instance as Bob, and I'm going to get prompted.

Let's take a look at what the configuration looks like here. I mentioned earlier that we've configured this instance with that Vault SSH helper. That's installed in /usr/local/bin.

There's the SSH helper itself, but let's take a look at the configuration. Alright, sshd_config.

This is an Ubuntu 18 instance, so it's got a weird configuration. The sshd_config, there are only a couple options that we need to set here. That's the challenge-response authentication, and password authentication is no, so we've set it to no. I have all of this documented, and using PAM as yes. We're going to use PAM, and I'm going to show you the PAM config here in a bit.

I've got all this on GitHub as well, it's all open-source. If you want to take a look, it's just at my HashiConf 2018 project. Errygg is my GitHub username. You can take a look and run this yourself.

Let's go back to pam.d, and look at the SSHD_config. This is where all the meat is, right here. This first line here, the auth, what this is doing is it's taking the exec for when you actually go and start to SSH into your instance. It's delegating that now to the Vault SSH helper.

The configuration is set to access the Vault instance. The role is configured in there. Only vampires are allowed to use this. Let's take a look at the configuration here.

The allowed roles here are vampires. The allowed CIDR list is everything, and then that's the Vault address I'm currently using. As a caveat here, this is not a production system, so we're not using TLS certificates right now to secure this. You'll notice the HTTP versus HTTPS, but this gives you an idea of how the configuration looks. If you think about it, it's the SSHD config, the PAM config, the Vault SSH helper binary, as well as the configuration.

You can think about how this was set up. I set it up using simple user data and some bash scripts, but you can use Chef, you can use Packer to build this into your images. There are a variety of different ways to push all this information to your instance, but this is how it can be set up to be able use a one-time password.

This right here is set up with users that are local to the box itself. Bob could be Ubuntu, Bob could be whoever you want it to be.

Okay, that's Bob. Let's take a look at Suzy now. Suzy's a little bit different, she's using a certificate authority. Here, same thing, we're just going to log in to Vault, same way. We're using the userpass method to be able to SSH into it. We could use LDAP, certainly. There are a lot of different authentication methods out there.

Let's make sure we've got a token. We got a token and an address. We're logged in to Vault now. Here we're going to write, similar to what we did in one-time password, we're going to write to Vault and we're going to ask it for a signed key.

We're going to get back a certificate from there. We're going to put it into our SSH directory. What this is going to allow us to do is SSH into the instance with that public key. What's kind of neat is that you can just throw this into your SSH directory and OpenSSH will recognize this file as an authority for you to be able to SSH into an instance.

I don't have to pass in any sort of flags or anything to be able to SSH into my instance. We're going to SSH as Suzy. Back to Terraform Enterprise, this is our instance here that we're going to SSH into.

Oh, but we got denied. Why did we get denied? We had a 30-second time-out, so that's why. Because it's a 30-second time-out, I wasn't able to get into this instance because I wasn't fast enough. Let's try it again and we'll see that you can SSH in.

Somebody streaming right now, I see you out there. Alright, cool, let's go ahead and SSH to the instance, and we're in.

In summary, this is the SSH backend for Vault. We can use one-time passwords. We can also use certificate authorities for those, for SSHing into your instances. With respect to the certificate authority, there's very little configuration that has to be done.

There are two different things that need to be done. There's a small configuration in your SSHD config. At the very bottom, we need to say that there's a trusted user CA that we use. The other configuration that needs to be put here is the actual key itself.

This is the certificate authority that we built prior. This happened before this talk, that I created the PEM key from Vault and pushed it to the instance, just using user data.

The other cool thing is that there's an endpoint, because it's just a public key. You can set in your user data, or Chef scripts, or whatever to pull those and put them onto your instances, just using that endpoint.

This is just one piece of the Vault Swiss Army knife, but it's nice to be able to use it for your instances. Especially if you're having to use SSH in your environment.

I will, in about 45 minutes or so, I'll be at the Vault booth if you want to get a little bit more of a scoop of how this was put together and how it works. How the Vault side of things is set up, versus the client side.

I think that's about it and thank you very much. I appreciate your time.