Manage Kubernetes Secrets for Flux with HashiCorp Vault
Configure the Secrets Store CSI driver with HashiCorp Vault to securely inject secrets into Flux or other GitOps tools on Kubernetes.
When you use GitOps to deploy your services to Kubernetes, you need a way for services to securely access secrets such as database credentials, deploy tokens, or certificates. GitOps works by using Git as the single source of truth for declarative infrastructure and applications. When you first implement GitOps, you may start by encrypting and committing your secrets to version control and enable a GitOps tool to decrypt them. For example, Flux natively supports Vault through SOPS.
However, encrypting and committing your secrets to version control comes with the risk of having to remediate compromised secrets if you accidentally commit them unencrypted. This approach also does not scale as your development teams, Kubernetes clusters, applications, and infrastructure grow. How do you dynamically inject secrets into your services deployed through GitOps?
This post will show two approaches to configuring a Kubernetes application deployed by the Flux framework to use secrets from HashiCorp Vault. When you use Kubernetes secrets, you have the additional management overhead of distributing and auditing hundreds of secrets across different namespaces and setting up role-based access control (RBAC) to limit access to secrets. Instead, you can use a file-based approach to injecting secrets by adding annotations for the Vault Agent sidecar injector or defining a volume for the Vault Container Storage Interface (CSI) provider.
However, you may have applications such as Flux that depend on Kubernetes secrets and cannot use the Vault Agent sidecar injector or CSI provider. This post introduces a second approach, in which the Vault CSI provider synchronizes secrets from Vault to a Kubernetes secret, allowing applications to reference the secret without refactoring. The approach offers an intermediate solution to let Kubernetes deployments migrate to using Vault while minimizing refactoring impact and integrating with tools that depend on Kubernetes secrets.
»Recommended Approach: File-Based Secrets Injection
In general, avoid using unencrypted, plaintext Kubernetes secrets. Instead, use a file-based approach to retrieve dynamic secrets. A file-based-approach limits access to secrets to the application’s container and service account. Writing the secret to a file in a container volume removes the need to manage separate Kubernetes RBAC permissions for different secrets and automatically updates the secret when you rotate it in Vault. The approach reduces potential errors resulting from incorrect RBAC permissions or revoked secrets.
Our recommended Vault integration approach writes secrets to a file in a volume mount using the Vault Agent sidecar injector or CSI provider. For more information on the differences between these methods, review our blog post comparing Kubernetes-Vault Integration via Sidecar Agent vs. CSI Provider.
With a file-based approach, you must refactor Kubernetes manifests for your applications to use these methods. After refactoring the application’s Kubernetes manifest for a file-based secrets injection approach, you can deploy the applications normally with GitOps tools like Flux. You do not need to configure Flux to retrieve secrets from Vault, as the application will automatically authenticate and retrieve the secret when it starts.
»Vault Agent Sidecar Injector
For the Agent sidecar injector, annotate your deployments with the path to the Vault credentials and a template to write a file with secrets.
The following example uses Vault annotations to inject a sidecar for a product-api. The Vault Agent sidecar will write the database credentials to a local file in a container volume called conf.json. The application container reads the file from /vault/secrets/conf.json.
apiVersion: apps/v1
kind: Deployment
metadata:
name: product-api
labels:
app: product-api
spec:
replicas: 1
selector:
matchLabels:
app: product-api
template:
metadata:
labels:
app: product-api
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "product"
vault.hashicorp.com/agent-inject-secret-conf.json: "hashicups/database/creds/product"
vault.hashicorp.com/agent-inject-template-conf.json: |
{
"bind_address": ":9090",
"metrics_address": ":9103",
{{ with secret "hashicups/database/creds/product" -}}
"db_connection": "host=postgres port=5432 user={{ .Data.username }} password={{ .Data.password }} dbname=products sslmode=disable"
{{- end }}
}
spec:
serviceAccountName: product-api
containers:
- name: product-api
image: hashicorpdemoapp/product-api:v0.0.20
ports:
- containerPort: 9090
- containerPort: 9103
env:
- name: "CONFIG_FILE"
value: "/vault/secrets/conf.json"
»Vault CSI Provider
For the Vault CSI provider, you must have the Secrets Store CSI driver installed on your Kubernetes cluster. Follow the instructions in our Learn tutorial to install and configure the Vault CSI provider.
Define a SecretProviderClass for the Vault CSI provider to retrieve secrets from Vault.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: products-db
spec:
provider: vault
parameters:
roleName: 'product'
vaultAddress: 'http://vault:8200'
objects: |
- objectName: "db-username"
secretPath: "hashicups/database/creds/product"
secretKey: "username"
- objectName: "db-password"
secretPath: "hashicups/database/creds/product"
secretKey: "password"
Add the SecretProviderClass as a volume mount for the deployment. The volume contains one file, called username, with the database username. It contains another file, called password, with the database password.
kind: Deployment
metadata:
name: product-api
labels:
app: product-api
spec:
replicas: 1
selector:
matchLabels:
app: product-api
template:
metadata:
labels:
app: product-api
spec:
serviceAccountName: product-api
containers:
- name: product-api
image: hashicorpdemoapp/product-api:v0.0.20
ports:
- containerPort: 9090
- containerPort: 9103
volumeMounts:
- name: products-db
mountPath: '/mnt/secrets-store'
readOnly: true
volumes:
- name: products-db
csi:
driver: 'secrets-store.csi.k8s.io'
readOnly: true
volumeAttributes:
secretProviderClass: products-db
»File-Based Approach with Environment Variables
If your application uses environment variables, you can refactor its Kubernetes manifest to source environment variables from a file. For example, the following database deployment includes a command to source an environment variable storing the database password.
apiVersion: apps/v1
kind: Deployment
metadata:
name: expense-db-mysql
labels:
app: expense-db-mysql
spec:
replicas: 1
selector:
matchLabels:
app: expense-db-mysql
template:
metadata:
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "expense-db-mysql"
vault.hashicorp.com/agent-inject-secret-db: "expense/static/data/mysql"
vault.hashicorp.com/agent-inject-template-db: |
{{ with secret "expense/static/data/mysql" -}}
export MYSQL_ROOT_PASSWORD="{{ .Data.data.db_login_password }}"
{{- end }}
labels:
app: expense-db-mysql
spec:
serviceAccountName: expense-db-mysql
containers:
- name: expense-db-mysql
image: "joatmon08/expense-db:mysql-8"
ports:
- containerPort: 3306
command: ["/bin/bash"]
args: ["-c", "source /vault/secrets/db && /usr/local/bin/docker-entrypoint.sh mysqld"]
With file-based secrets injection, you localize access to the secret to the application deployment. You avoid committing secrets (even encrypted) to version control and let a secrets manager handle the rotation of the secret. On the other hand, you must refactor your application’s Kubernetes manifest to reference the file containing the secret. Your application must also enable a reload capability to use the updated secrets file when Vault rotates the credentials.
»Alternative Approach: Sync as Kubernetes Secrets
If you use GitOps frameworks to deploy your applications and cannot refactor your application to use the file-based approach, you can use the Vault CSI provider to synchronize a Vault secret to a Kubernetes secret. Store your static credentials (such as a repository token) or set up dynamic credentials (such as a database username) in Vault and use the Vault CSI provider to write them to a Kubernetes secret.
With this approach, you do not need to refactor your application’s Kubernetes manifest. Furthermore, this approach automatically handles any updates you make to secrets in Vault and synchronizes the changes to a Kubernetes secret. However, make sure to control access to the Kubernetes secret with RBAC policies. This prevents unauthorized applications or users from retrieving the unencrypted secret.
To start using the Vault CSI provider, you need the Secrets Store CSI driver installed on your Kubernetes cluster. Follow the instructions in our Learn tutorial to install and configure the Vault CSI provider. You will also need to set up the Kubernetes authentication method in your Vault cluster to allow authentication by Kubernetes service accounts.
»Example: Injecting GitLab Deploy Tokens to Flux
The example described here synchronizes a GitLab deploy token for a private repository to a Kubernetes secret using the Vault CSI provider. Flux can reference the Kubernetes secret. Similarly, you can extend this workflow to synchronize secrets from any Vault secrets engine to any applications that references Kubernetes secrets as files or environment variables. You can manage static secrets like a GitLab deploy token for a private repository in Vault and automatically sync updates to a Kubernetes secret using the Vault CSI provider. Flux needs access to the deploy token in order to synchronize with a private repository. This example uses HashiCorp Terraform to configure GitLab and Vault.
Use the GitLab provider for Terraform to create a GitLab deploy token for your private GitLab project (repository).
resource "gitlab_deploy_token" "hashicups" {
depends_on = [gitlab_project.hashicups]
project = gitlab_project.hashicups.id
name = "Deploy Token for Flux"
username = local.flux_username
scopes = ["read_repository", "read_registry"]
}
Then, store the GitLab username and token as a static secret in Vault’s key-value secrets engine. Pass the token into the password field. Flux needs username and password data to access a private repository.
resource "vault_mount" "flux" {
path = "hashicups/flux"
type = "kv-v2"
description = "For hashicups Flux deploy secrets"
}
resource "vault_generic_secret" "gitlab" {
path = "${vault_mount.flux.path}/gitlab"
data_json = jsonencode({
username = "${gitlab_deploy_token.hashicups.username}"
password = "${gitlab_deploy_token.hashicups.token}"
})
}
Define a Vault role with the Vault provider for Terraform. The Vault role allows Flux’s source-controller service account in the flux-system namespace to retrieve the username and password for the private repository. This ensures that Flux can read the secret but not change it.
data "vault_policy_document" "gitlab" {
rule {
path = "hashicups/flux/data/gitlab"
capabilities = [
"read"
]
description = "read GitLab credentials"
}
}
resource "vault_policy" "gitlab" {
name = "gitlab"
policy = data.vault_policy_document.gitlab.hcl
}
resource "vault_kubernetes_auth_backend_role" "gitlab" {
backend = vault_auth_backend.kubernetes.path
role_name = "flux"
bound_service_account_names = ["source-controller"]
bound_service_account_namespaces = ["flux-system"]
token_ttl = 3600
token_policies = [vault_policy.gitlab.name]
}
After setting up the static secret in Vault, deploy a SecretProviderClass in the flux-system namespace to allow the Vault CSI provider to retrieve it from Vault. Unlike the previous example for SecretProviderClass, this definition includes the secretObjects field. secretObjects writes the username and password to a Kubernetes secret called gitlab-credentials with type Opaque. Our example uses the flux Vault role to retrieve the GitLab repository’s username and deploy token.
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: gitlab-credentials
namespace: flux-system
spec:
provider: vault
secretObjects:
- secretName: gitlab-credentials
type: Opaque
data:
- objectName: gitlab-username
key: username
- objectName: gitlab-password
key: password
parameters:
roleName: 'flux'
vaultAddress: 'http://vault:8200'
objects: |
- objectName: "gitlab-username"
secretPath: "hashicups/flux/data/gitlab"
secretKey: "username"
- objectName: "gitlab-password"
secretPath: "hashicups/flux/data/gitlab"
secretKey: "password"
However, the Secrets Store CSI driver does not create the secret until a deployment creates a volume mount. Create a deployment that mounts the Secrets Store CSI volume. The deployment creates a Kubernetes secret called gitlab-credentials when it mounts the volume. Make sure the deployment uses the same serviceAccountName (source-controller) and namespace (flux-system) configured in the Vault role.
apiVersion: apps/v1
kind: Deployment
metadata:
name: secrets-store-inline
namespace: flux-system
labels:
app: secrets-store-inline
spec:
replicas: 1
selector:
matchLabels:
app: secrets-store-inline
template:
metadata:
labels:
app: secrets-store-inline
spec:
serviceAccountName: source-controller
containers:
- name: busybox
image: k8s.gcr.io/e2e-test-images/busybox:1.29
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "gitlab-credentials"
The deployment can use any container image compliant with your organization and use limited resources. In general, it helps to create a separate deployment that initializes and continuously synchronizes the secret. This means tools and applications can reference the Kubernetes secret as an abstraction between Vault and Flux. Flux does not have to read the secret directly from Vault.
Check if the Secrets Store CSI driver created a Kubernetes secret called gitlab-credentials in the flux-system namespace. It should have two data entries.
$ kubectl get secrets gitlab-credentials -n flux-system
NAME TYPE DATA AGE
gitlab-credentials Opaque 2 4h14m
Use this private repository username and token to set up a GitRepository in Flux. Update secretRef to reference the gitlab-credentials Kubernetes secret.
apiVersion: source.toolkit.fluxcd.io/v1beta1
kind: GitRepository
metadata:
name: hashicups
namespace: flux-system
spec:
interval: 1m0s
ref:
branch: main
url: https://gitlab.com/joatmon08/hashicups.git
secretRef:
name: gitlab-credentials
Any Kustomization referencing the GitRepository source should successfully log into the GitLab private repository and deploy Kubernetes resources.
You can apply the Flux GitRepository example to retrieve secrets from any Vault secrets engine for any application. Your applications and object manifests can keep using references to Kubernetes secrets and benefit from Vault managing the secrets. Setting up the Vault CSI driver with synchronization to Kubernetes secrets minimizes the disruption of migrating your application’s secrets to a secrets manager. Later, you can gradually refactor the application to use file-based secrets injection.
»Conclusion
When possible, use a file-based approach to inject secrets from Vault into Kubernetes deployments. You can use the Vault Agent sidecar injector or CSI provider to read secrets from Vault. Both of them write the secret to a file on a container volume, which removes the need to use Kubernetes RBAC to secure access to the secret and handles dynamic credentials.
When should you consider synching a Vault secret as Kubernetes secret with the Vault CSI provider? Do it when you need to use Kubernetes secrets for compatibility with a GitOps tool like Flux, or if you just want to minimize the impact of refactoring applications. By implementing a background deployment to mount the CSI volume, you allow the controller to continuously read secrets from Vault and update a Kubernetes secret. Secure access to the Kubernetes secret and Secrets Store CSI driver objects with Kubernetes RBAC.
If you would like to review the configuration in this blog post, check out the code repository for this post. For additional information on deploying Vault on Kubernetes, review our Learn tutorial. Get started on file-based secrets injection with tutorials on the Vault agent sidecar injector and CSI provider. Learn more about Flux here to deploy your applications in a GitOps manner.
Questions about this post? Add them to the community forum!
Sign up for the latest HashiCorp news
More blog posts like this one
Vault integrations with MongoDB, Private Machines, and walt.id strengthen customer security
Three new HashiCorp Vault ecosystem integrations extend security use cases for customers.
HashiCorp at re:Invent 2024: Security Lifecycle Management with AWS
A recap of HashiCorp security news and developments on AWS from the past year, for your security management playbook.
HCP Vault Dedicated adds secrets sync, cross-region DR, EST PKI, and more
The newest HCP Vault Dedicated 1.18 upgrade includes a range of new features that include expanding DR region coverage, syncing secrets across providers, and adding PKI EST among other key features.