Bridgewater: Securing their AWS Infrastructure with Vault

This is a guest post by Joel Thompson, Systems Engineer at Bridgewater Associates. Joel is a user of Vault at Bridgewater Associates and a contributor to the HashiCorp Vault project, specifically for the AWS IAM Authentication method discussed in this post.

HashiCorp released Vault 0.7.1 which ships with a major enhancement to the AWS-EC2 authentication backend, now renamed to the AWS authentication backend, making it easy for many different AWS resource types to securely authenticate with Vault and get a Vault token. Lambda functions, ECS jobs, EC2 instances, or any other client with access to AWS IAM credentials can use those credentials to securely authenticate to Vault to retrieve their secrets. I'm really excited about the feature, and I think it'll be a game changer for all the security-conscious AWS customers out there.

First, though, some introductions. Bridgewater Associates is focused on understanding how the world works. By having the deepest possible understanding of the global economy and financial markets, and translating that understanding into great portfolios and strategic partnerships with institutional clients, we've built a distinct track record of success. Today, we manage about $160 billion for approximately 350 of the largest and most sophisticated global institutional clients including public and corporate pension funds, university endowments, charitable foundations, supranational agencies, sovereign wealth funds, and central banks.

A few years ago, Bridgewater started moving to Amazon Web Service's cloud offering, and I was one of the first engineers involved with that effort. We loved many of the advantages offered by AWS, but there was one problem that had consistently caused us pain. How do we take advantage of the dynamic compute capabilities offered by a cloud provider without sacrificing security? Or, to put it concretely, how do we securely grant new instances in an AWS autoscaling group access to secrets they need?

This was clearly a need that others in the community shared. There were some suggestions from the community about how to achieve it, including two from one of my coworkers, Dan Peebles.

In response to this need, HashiCorp released the AWS-EC2 authentication backend. The AWS-EC2 authentication backend was great for what was feasible at the time, but it had limitations -- in particular, it only worked for EC2 instances and not AWS Lambda functions or ECS jobs or other AWS scenarios. And then things changed when AWS added the sts:GetCallerIdentity method. Suddenly, the proposal in GH-948 felt like it was within reach. As someone who loves AWS, loves security, and loves HashiCorp, implementing that proposal and making it real was a great opportunity for me to contribute something meaningful back to the community. After collaborating closely with the HashiCorp Vault engineering team, and getting help from Dan (who originally suggested GH-948, the AWS authentication backend was born.

The AWS authentication backend now has the concept of authentication types and currently supports two. The EC2 authentication type corresponds to the prior behavior of the AWS-EC2 authentication backend, while the IAM authentication type is the new method and it allows you to authenticate using AWS IAM credentials, mapping an IAM user or role to a Vault role. In many cases, AWS already does the hard work of securely providing your compute resources with IAM credentials, such as EC2 instances in an instance profile, AWS Lambda functions, ECS jobs, and AWS CodeBuild steps. AWS has done the hard work of providing your resources with IAM credentials, and now your applications can just consume these credentials to authenticate to Vault in order to access their secrets. Vault authentication is performed by having clients sign an AWS API request, sts:GetCallerIdentity. API request and then send the signed request to Vault. The actual AWS secret key is never sent to Vault, helping you keep your sensitive credentials safe. Vault forwards the signed request on to AWS and observes the result; if the result matches the entity configured in a Vault role, then the request is authenticated and a Vault token is returned to the client.

This also solves another common problem: how to make the developer workflow match the production workflow as much as possible. Many developers want to test locally on their own laptops, which aren't able to readily authenticate to Vault using the EC2 authentication method. However, with the new IAM authentication method, clients can consume developers' AWS API credentials to authenticate to Vault, providing the same workflow as in your production environments.

The IAM authentication method also supports a feature called "inferencing." The EC2 authentication method allows you to place restrictions on EC2 instances that can authenticate to a role, such as which AMI an instance was launched from, or which subnet the instance was in. Because these are attributes about an EC2 instance and not an IAM principal, similar sorts of restrictions aren't generally possible with the IAM authentication method. However, EC2 instances in an IAM instance profile authenticate in a way that makes their instance IDs visible to Vault. Inferencing tells Vault to look for the instance ID, look it up in EC2, and treat the client as an EC2 instance, allowing you to apply many of the EC2-specific binds as when using the EC2 auth method. (Note that there are some very important security caveats you should be aware of before just blindly trusting inferencing which are described in the documentation.)

The vault command-line tool makes it easy to authenticate to Vault using the AWS authentication backend. Just run:

$ vault auth -method=aws role=vault_role_name

This will automatically look up AWS credentials from the standard locations defined by AWS (environment variables, credential file, or IAM instance profile) and use those to authenticate to Vault. The CLI also accepts an access key and secret key on the command line if necessary, but you should avoid putting sensitive credentials on the command line if at all possible. For programmatic use cases, you should look at the AWS Signature v4 documentation and the source code of the Vault CLI tool for an example of how to generate these values and pass them to Vault. Integrating the new IAM authentication method with community-supported Vault clients is something we're looking forward to helping the community implement; if you're interested in contributing an integration, please reach out to the community for help or advice if needed.

I'm proud to have had a hand in creating something that I'm confident members of the community will find tremendously valuable. I would also like to acknowledge and thank the engineers from the HashiCorp Vault team, Jeff and Vishal, and my co-worker, Dan, for their help and patience in working with me on this. Without their assistance, this wouldn't be a reality.

HashiCorp Vault is a product to secure any infrastructure and any application. Vault provides a consistent workflow to securely store and manage secrets, provide encryption as a service, and granular privilege access management. To learn more about HashiCorp Vault visit https://www.hashicorp.com/products/vault/.