Injecting Vault Secrets Into Kubernetes Pods via a Sidecar

This guide exists for historical purposes, but a more up-to-date guide can be found on the HashiCorp Developer page: Injecting Secrets into Kubernetes Pods via Vault Helm Sidecar.

We are excited to announce a new Kubernetes integration that enables applications with no native HashiCorp Vault logic built-in to leverage static and dynamic secrets sourced from Vault. This is powered by a new tool called vault-k8s, which leverages the Kubernetes Mutating Admission Webhook to intercept and augment specifically annotated pod configuration for secrets injection using Init and Sidecar containers.

Applications need only concern themselves with finding a secret at a filesystem path, rather than managing tokens, connecting to an external API, or other mechanisms for direct interaction with Vault.

Here are a few supported use-cases:

• Init only container to pre-populate secrets before an application starts. For example, a backup job that runs on a regular schedule and only needs an initial secret at start time.
• Init and Sidecar. Init container to fetch secrets before an application starts, and a Sidecar container that starts alongside your application for keeping secrets fresh (sidecar periodically checks to ensure secrets are current). For example, a web application that is using dynamic secrets to connect to a database with an expiring lease.
• Pod authentication through Kubernetes Service Account for Vault Policy enforcement. For example, you likely want to restrict a Pod to only access the secrets they need to function correctly. This should also assist in auditing secret usage of each application.
• Flexible output formatting options using the Vault Agent template functionality which was incorporated from consul-template. For example, fetching secret data from Vault to creating a database connection string, or adapting your output to match pre-existing configuration file formats, etc.

Our continuing goal is to expand Kubernetes support and give you a variety of options around how you can leverage Vault to securely introduce secrets into your workflow. You can learn more about our thinking here by reading our What's Next for Vault and Kubernetes blog post.

»Video Walk Through

To see a video demo of Vault secrets being injected into Kubernetes pods using init and sidecar containers please watch the video below. We will walk through the vault-k8s initial setup using the Vault Helm Chart and cover three example use-cases (adding annotations, output formatting, and background jobs). The video should help round out your understanding of how this works in practice.

»How it works

The recommended installation method is through the latest Vault Helm Chart which now supports the vault-k8s injection functionality (see documentation). A Docker image is also available. The Docker image can be used to manually run vault-k8s within your scheduled environment if you choose not to use the Helm Chart. For this blog, the focus is on using the Vault Helm Chart, as that is likely a good starting point for learning about this feature.

The Helm Chart, with the injection feature enabled, launches Vault, along with the vault-k8s injector service and registers itself with Kubernetes as a Mutating Admission Webhook (tied to a specific namespace). The diagram below illustrates how the vault-k8s webhook is used to intercept and change pod configuration when a Kubernetes API request is made.

Diagram inspired by the Guide to Kubernetes Admission Controllers.

You can opt each application into Vault secret injection through the use of specifically set annotations within the pod configuration. Then, when the vault-k8s webhook detects these specific annotations, it rewrites the pod definition based on what was requested (through your set annotations). Think of these annotations as configuration parameters for how you want secrets to be presented inside your application containers. You can leverage a selected namespace, specifically set annotations, and Kubernetes Service Accounts tied to a Vault Policy, this gives you fine-grained control of where and what secrets are injected without compromising on security.

So, what do these Vault specific pod annotations look like? You can check out the documentation for a complete listing, but we’ll cover a few examples below that should give you a good idea. You can use the kubectl patch command to apply the annotations to an existing Pod object, these will be intercepted by the vault-k8s webhook service, which will then inject the correct init and sidecar containers along with the requested secrets (if you have access based on the Service Account and associated Vault Policy).

# patch-basic-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secrets/helloworld"
        vault.hashicorp.com/role: "myapp"

For example, the above annotations when applied instructs vault-k8s to inject an init and sidecar container into the requested Pod, fetch the secret/helloworld secret from Vault, and populate /vault/secrets/helloworld with that data, if the “myapp” role has access.

»Secrets Injection Workflow

In this section we are going to work through an end-to-end secrets injection workflow for getting started with vault-k8s using two examples. As mentioned above, the recommended installation method is the official Vault Helm Chart. The Chart, with the Agent Sidecar Injection feature enabled, launches Vault, the vault-k8s webhook Injector web service, and configure the Kubernetes Mutating Admission Webhook.

First, before we install Vault, make sure injector support is enabled in the Vault Helm Chart values.yaml file.

injector:
  enabled: true

Next, let’s configure a demo namespace, set the current context to it, and install Vault using the Helm Chart.

kubectl create namespace demo
kubectl config set-context --current --namespace=demo

helm install --name=vault \
       --set='server.dev.enabled=true' \
       ./vault-helm

Next, connect to Vault and configure a policy named “app” for the demo. This is a very non-restrictive policy, and in a production setting, you would typically want to lock this down more, but it serves as an example while you play around with this feature.

kubectl exec -ti vault-0 /bin/sh

cat <<EOF > /home/vault/app-policy.hcl
path "secret*" {
  capabilities = ["read"]
}
EOF

vault policy write app /home/vault/app-policy.hcl

Next, we want to configure the Vault Kubernetes Auth method and attach our newly recreated policy to our applications service account (we’ll create the application in just a minute).

vault auth enable kubernetes

vault write auth/kubernetes/config \
   token_reviewer_jwt="$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)" \
   kubernetes_host=https://${KUBERNETES_PORT_443_TCP_ADDR}:443 \
   kubernetes_ca_cert=@/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

vault write auth/kubernetes/role/myapp \
   bound_service_account_names=app \
   bound_service_account_namespaces=demo \
   policies=app \
   ttl=1h

Finally, let's create an example username and password in Vault using the KV Secrets Engine. The end goal here, is for this username and password to be injecting into our target pod's filesystem, which knows nothing about Vault.

vault kv put secret/helloworld username=foobaruser password=foobarbazpass

Here is an example app.yaml configuration file for running a demo application. This spawns a simple web service container useful for our testing purposes. We are also defining a Service Account which we can then tie back to the Vault Policy we created earlier. This allows you to specify each secret an application is allowed to access.

# app.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app
  labels:
    app: vault-agent-demo
spec:
  selector:
    matchLabels:
      app: vault-agent-demo
  replicas: 1
  template:
    metadata:
      annotations:
      labels:
        app: vault-agent-demo
    spec:
      serviceAccountName: app
      containers:
      - name: app
        image: jweissig/app:0.0.1
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: app
  labels:
    app: vault-agent-demo

Next, lets launch our example application and create the service account. We can also verify there are no secrets mounted at /vault/secrets.

kubectl create -f app.yaml
kubectl exec -ti app-XXXXXXXXX -c app -- ls -l /vault/secrets

Here is a annotations patch we can apply to our running example application’s pod configuration that sets specific annotations for injecting our secret/helloworld Vault secret.

# patch-basic-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/role: "myapp"

Next, let’s apply our annotations patch.

kubectl patch deployment app --patch "$(cat patch-basic-annotations.yaml)"
kubectl exec -ti app-XXXXXXXXX -c app -- cat /vault/secrets/helloworld

If all goes correctly you should now see a tmpfs mount at /vault/secrets and a helloworld secret containered in there. What happened here, is that when we applied the patch, our vault-k8s webhook intercepted and changed the pod definition, to include an Init container to pre-populate our secret, and a Vault Agent Sidecar to keep that secret data in sync throughout our applications lifecycle.

However, if you were to actually run this, you would notice that the data in /vault/secrets/helloworld is not formatted, and is just a Go struct export of our secret. This can be problematic, since you almost certainly want the output to be formatted in a particular way, for consumption by your application. So, what is the solution?

data: map[password:foobarbazpass username:foobaruser]
metadata: map[created_time:2019-12-16T01:01:58.869828167Z deletion_time: destroyed:false version:1]

Well, you can format your secret data using by leveraging Vault Agent Templates, which is very useful for dealing with your various output formatting needs. In the next example here, we are going to parse our secrets data into a postgresql connection string. Templates can come in extremely handy and should fit a variety of your use cases, where you need to conform to existing configuration formats, construct connection stings (as seen below), etc.

# patch-template-annotations.yaml
spec:
  template:
    metadata:
      annotations:
        vault.hashicorp.com/agent-inject: "true"
        vault.hashicorp.com/agent-inject-status: "update"
        vault.hashicorp.com/agent-inject-secret-helloworld: "secret/helloworld"
        vault.hashicorp.com/agent-inject-template-helloworld: |
          {{- with secret "secret/helloworld" -}}
          postgresql://{{ .Data.data.username }}:{{ .Data.data.password }}@postgres:5432/wizard
          {{- end }}
        vault.hashicorp.com/role: "myapp"

Next, just like before, we apply the annotations patch.

kubectl patch deployment app --patch "$(cat patch-template-annotations.yaml)"
kubectl exec -ti app-XXXXXXXXX -c app -- cat /vault/secrets/helloworld

With the output looking like this:

postgresql://foobaruser:foobarbazpass@postgres:5432/wizard

The above examples covered a pretty straightforward workflow pattern for injecting a secret into a running application that has no native Vault logic built-in. The applications only need to concern themselves with finding a secret at a filesystem path, that’s it.

»Next Steps

The vault-k8s repo is available now on GitHub. To learn more, please review the Agent Sidecar Injector documentation and watch our Injecting Vault Secrets Into Kubernetes Pods via a Sidecar webinar. There is also a new guide published on HashiCorp Learn and a new blog from our Solutions Engineering team that dives even further into the sidecar injection use case.

Also, if you enjoy playing around with this type of stuff, maybe you’d be interested in working at HashiCorp too since we’re hiring!