Cloud Foundry Vault Service Broker

We are pleased to announce the release of the official Cloud Foundry HashiCorp Vault Service Broker. This service broker connects to an existing Vault cluster and can be used by multiple tenants within Cloud Foundry to securely store, access, and encrypt using Vault.

»Why Vault & Cloud Foundry Integration

One of the core components of HashiCorp is our commitment to heterogeneous environments. The Cloud Foundry HashiCorp Vault Service Broker is an example of our continued commitment to work with our ecosystem partners to deliver the best experience to our users regardless of their infrastructure or application type. In addition, the Tao of HashiCorp showcases simplicity and modularity combined with communication of sequential processes. These key pillars drive us to create pluggable components. Simply put:

You don't have to use all the HashiCorp tools to get the benefits of one.

You should not have to re-architect your entire infrastructure just to use a single tool. Vault is a tool for managing secrets, and while we would love you to deploy it with Terraform and run it with Nomad, you don't have to. Our tools function in isolation to build workflows that enable you to provision, secure, and run any infrastructure for any application.

»Background

Cloud Foundry provides a framework for deploying and managing applications. Those applications have the ability to communicate with each other, or with pre-configured services. These services expose APIs via service brokers, and can provide data to an application including credentials, endpoints, or application-specific data.

When a new service instance is provisioned against the Cloud Foundry HashiCorp Vault Service Broker, it will perform the following operations in the Vault server:

1. Mount the generic backend at /cf/<organization_id>/secret/
2. Mount the generic backend at /cf/<space_id>/secret/
3. Mount the generic backend at /cf/<instance_id>/secret/
4. Mount the transit backend at /cf/<instance_id>/transit/

The mount operation is idempotent, so service instances in the same organization or space will not re-create the mount. These mount points are returned to the application as part of the credential, so there is no need for applications to guess or interpolate these strings.

The read-only organization mount allows for sharing secrets organization wide, and the read-write mount permits sharing secrets between applications in the same Cloud Foundry Space via the generic backend. The transit backend provides "encryption-as-a-service" on a per-application basis.

We believe this architecture provides the best solution for applications to share secrets between instances, with other applications in the same space, and retrieve read-only information from the organization in Cloud Foundry. The beauty of this design is that applications do not need to be Vault-aware to gain the benefits of modern secrets management.

»Getting Started

The Cloud Foundry HashiCorp Vault Service Broker assumes a Vault server is already-configured. This blog post denotes the address and token using the following variables on the local workstation:

$ export VAULT_ADDR="https://vault.company.internal/"
$ export VAULT_TOKEN="abcdef-134255-..."

The Cloud Foundry HashiCorp Vault Service Broker does not run or configure a Vault server. This Vault server above does not need to be running under Cloud Foundry, but it must be accessible from within Cloud Foundry or wherever the broker is deployed.

Additionally the broker is configured to use basic authentication. Those variables will be denoted as follows in this post:

$ export USERNAME="vault"
$ export PASSWORD="vault"

Additionally, this post assumes a Cloud Foundry cluster is already up and running. To start a cluster locally, we recommend using PCF Dev.

»Deploying the Broker

The first step is deploying the broker. This broker can run anywhere including Cloud Foundry, Heroku, HashiCorp Nomad, or your local laptop. This example shows running the broker under Cloud Foundry.

First, create a space in which to run the broker:

$ cf create-space vault-broker

Switch over to that space:

$ cf target -s vault-broker

Deploy the cf-vault-service-broker by cloning the repository from GitHub:

$ git clone https://github.com/hashicorp/cf-vault-service-broker
$ cd cf-vault-broker

And push to Cloud Foundry:

$ cf push --random-route --no-start

• The --random-route flag is optional, but it allows us to easily run more than one Vault broker if needed, instead of relying on "predictable names".

• The --no-start flag is important because we have not supplied the required environment variables to our application yet. It will fail to start now.

To configure the broker, provide the following environment variables:

$ cf set-env vault-broker VAULT_ADDR "${VAULT_ADDR}"
$ cf set-env vault-broker VAULT_TOKEN "${VAULT_TOKEN}"
$ cf set-env vault-broker SECURITY_USER_NAME "${USERNAME}"
$ cf set-env vault-broker SECURITY_USER_PASSWORD "${PASSWORD}"

Now that it is configured, start the broker:

$ cf start vault-broker

To verify the Cloud Foundry HashiCorp Vault Service Broker is running, execute:

$ cf apps

name           requested state   instances   memory   disk   urls
vault-broker   started           1/1         256M     512M   vault-broker-torpedolike-reexploration.local.pcfdev.io

Grab the URL and save it in a variable or copy it to your clipboard - we will need this later.

BROKER_URL=$(cf app vault-broker | grep urls: | awk '{print $2}')

Again, there is no requirement that our broker run under Cloud Foundry - this could be a URL pointing to any service that hosts the broker, which is just a Golang HTTP server.

To verify the broker is working as expected, query its catalog:

$ curl -s "${USERNAME}:${PASSWORD}@${BROKER_URL}/v2/catalog"

The result will be JSON that includes the list of plans for the broker:

{
	"services": [
		{
			"id": "0654695e-0760-a1d4-1cad-5dd87b75ed99",
			"name": "vault",
			"description": "HashiCorp Vault Service Broker",
			"bindable": true,
			"plan_updateable": false,
			"plans": [
				{
					"id": "0654695e-0760-a1d4-1cad-5dd87b75ed99.default",
					"name": "default",
					"description": "Secure access to a multi-tenant HashiCorp Vault cluster",
					"free": true
				}
			]
		}
	]
}

The HashiCorp Vault Service Broker is now running under Cloud Foundry and ready to receive requests.

»Register the Vault Broker

Before it can bind to services, the broker must be registered with Cloud Foundry. Remember that there is no requirement that the broker be running under Cloud Foundry, so we will need to provide the broker registration service the BROKER_URL from above. Again, this is important because at HashiCorp we believe in a vision of heterogeneity; you should not have to fit all your services into the perfect container to absorb modern best practices.

To register the broker, an application developer first creates a new space where they will request access to the broker:

$ cf create-space example
$ cf target -s example

Next, register the broker in this space:

$ cf create-service-broker vault-broker vault vault "https://${BROKER_URL}" --space-scoped

To verify the command worked, query the marketplace. You should see the broker in addition to the built-ins:

$ cf marketplace
# ...
vault          default           HashiCorp Vault Service Broker

»Generate Credentials through the Vault Broker

After registering the service in the marketplace, it is now possible to create a service instance and bind to it.

$ cf create-service vault default my-vault

Next, create a service key:

$ cf create-service-key my-vault my-key

And finally retrieve credentials for this instance:

$ cf service-key my-vault my-key

The response will look like:

{
	"address":"https://vault.company.internal:8200/",
	"auth":{
		"accessor":"b1074bb8-4d15-36cf-54dd-2716fb8ac91d",
		"token":"dff95895-6a03-0b29-6458-dc8602dc9df8"
	},
	"backends":{
		"generic":"cf/203f2469-04e4-47b8-bc17-f3af56df8019/secret",
		"transit":"cf/203f2469-04e4-47b8-bc17-f3af56df8019/transit"
	},
	"backends_shared":{
		"organization":"cf/3c88c61e-875b-4530-b269-970f926340c4/secret",
		"space":"cf/0348d384-f7d4-462b-9bd9-5a4c05b21b6c/secret"
	}
}

The keys are as follows:

• address - address to the Vault server to make requests against

• auth.accessor - token accessor which can be used for logging

• auth.token - token to supply with requests to Vault

• backends.generic - namespace in Vault where this token has full CRUD access to the static secret storage ("generic") backend

• backends.transit - namespace in Vault where this token has full access to the transit ("encryption as a service") backend

• backends_shared.organization - namespace in Vault where this token has read-only access to organization-wide data; all instances have read-only access to this path, so it can be used to share information across the organization.

• backends_shared.space - namespace in Vault where this token has read-write access to space-wide data; all instances have read-write access to this path, so it can be used to share information across the space.

»Conclusion

With Vault we are focused on providing the best possible distributed secrets store, regardless of application type. Because as infrastructure environments become increasingly distributed due to the inevitabilities of cloud, security as a foundational element of the application delivery process must be considered as a discrete concern. Graphically, our users tell us that they view their infrastructure along the lines of the graphic below, with Security as an element that must span the overall stack:

Organizations choose a runtime environment that best suits their needs, and those decisions span a vast array of technologies, either by design, default, opinion, or requirements of the applications. Heterogeneity is an important feature (not a bug) in the modern data center. This is the reason that Vault, like all the HashiCorp tools, is designed to support this heterogeneous environment. Because of this design decision, Vault is the one of the most widely used open source distributed secrets stores across so many organizations today. Vault focuses on workflows, not technologies, and this focus allows organizations to leverage Vault regardless of their size, scale, or technology choice.

While microservice and container adoption are key drivers for Vault, they are supported by a range of other technologies. Some of these technologies are at the bleeding edge, like Kubernetes and Nomad, but others are some of the most mature in the world, like the Spring Framework. With the new Cloud Foundry HashiCorp Vault Service Broker, security teams can safely extend Vault as the basis for secure secret management and encryption in Cloud Foundry.

Like most things we build, the Cloud Foundry HashiCorp Vault Service Broker is fully open source under the MPLv2 license and can be downloaded from the hashicorp/cf-vault-broker repository on GitHub.