Kubernetes secrets management with HCP Vault Secrets
Learn how to sync secrets from HCP Vault Secrets to Kubernetes with Vault Secrets Operator.
Most Kubernetes resources and workloads reference the Kubernetes Secret object for credentials, API tokens, certificates, and other confidential data. Kubernetes stores secrets unencrypted by default and requires role-based access control (RBAC) rules to ensure least-privilege access. However, it does not offer a straightforward method for tracking the lifecycle and distribution of the secret. Rather than store secrets in Kubernetes, you can use a centralized secrets management solution like HashiCorp Cloud Platform (HCP) Vault Secrets to audit and manage secrets.
This post demonstrates how to use the Vault Secrets Operator (VSO) to retrieve dynamic secrets from HCP Vault Secrets and write them to a Kubernetes Secret for other workloads and resources to reference. HCP Vault Secrets stores a set of secrets and supports the management and audit of their lifecycle and distribution. The demo app repository for this tutorial uses HashiCorp Terraform to write the secrets into HCP Vault Secrets, deploy the Vault Secrets Operator to the Kubernetes cluster, and deploy the custom resources to synchronize a secret for Argo CD to create a private repository. This workflow minimizes the need to refactor applications to access a secrets manager directly by using native Kubernetes Secrets.
»Store secrets in HCP Vault Secrets
HCP Vault Secrets enables you to manage the lifecycle of credentials and track their usage. For example, a GitOps tool like Argo CD requires credentials to access a private repository on GitHub. These credentials may include a private key, username and password, or token to allow Argo CD to read from a repository. If you set up the GitHub App, you need to store the application identifier and private key for Argo CD to use. Storing the credentials in HCP Vault Secrets and installing the Vault Secrets Operator ensures the credentials get synchronized with a Kubernetes Secret, which Argo CD will reference.
Workflow for Vault Secrets Operator & HCP Vault Secrets
The configuration example below uses the HCP provider for Terraform to store the GitHub App’s private key, application identifier, and application installation identifier in a HCP Vault Secrets application named argocd:
resource "hcp_vault_secrets_app" "argocd" {
app_name = "argocd"
description = "Secrets related to running Argo CD on Kubernetes"
}
resource "hcp_vault_secrets_secret" "argocd_github_app_id" {
app_name = hcp_vault_secrets_app.argocd.app_name
secret_name = "githubAppID"
secret_value = var.argocd_github_app.id
}
resource "hcp_vault_secrets_secret" "argocd_github_app_installation_id" {
app_name = hcp_vault_secrets_app.argocd.app_name
secret_name = "githubAppInstallationID"
secret_value = var.argocd_github_app.installation_id
}
resource "hcp_vault_secrets_secret" "argocd_github_app_private_key" {
app_name = hcp_vault_secrets_app.argocd.app_name
secret_name = "githubAppPrivateKey"
secret_value = base64decode(var.argocd_github_app.private_key)
}
resource "hcp_vault_secrets_secret" "argocd_github_url" {
app_name = hcp_vault_secrets_app.argocd.app_name
secret_name = "url"
secret_value = var.argocd_github_app.url
}
After applying the Terraform configuration, you can access the secrets in HCP Vault Secrets under the argocd application. You then need to synchronize these secrets from the argocd HCP Vault Secrets application into Kubernetes for Argo CD to reference.
Argo CD secrets in HCP Vault Secrets
»Install Vault Secrets Operator
Vault Secrets Operator helps synchronize secrets from HCP Vault Secrets or Vault into Kubernetes Secrets. The Operator handles the creation of custom resources that define the authentication to, and retrieval of, secrets from HCP Vault Secrets.
Here’s how to install Vault Secrets Operator using its Helm chart:
$ helm repo add hashicorp https://helm.releases.hashicorp.com
$ helm install -n vault-secrets-operator --create-namespace \
vault-secrets-operator hashicorp/vault-secrets-operator
The operator’s Helm chart includes a set of custom resource definitions for authenticating to HCP Vault Secrets.
»Authenticate to HCP Vault Secrets from Kubernetes
The Vault Secrets Operator needs a service principal to authenticate to HCP Vault Secrets. You can use Terraform to create a separate service principal with the viewer role (read-only access) that gives the operator read access to the secret:
resource "hcp_service_principal" "argocd" {
name = "argocd"
}
resource "hcp_service_principal_key" "argocd" {
service_principal = hcp_service_principal.argocd.resource_name
}
resource "hcp_project_iam_binding" "argocd" {
project_id = var.hcp_project_id
principal_id = hcp_service_principal.argocd.resource_id
role = "roles/viewer"
}
Save the HCP principal’s client ID and key to a Kubernetes Secret in the argocd namespace:
apiVersion: v1
data:
clientID: REDACTED
clientSecret: REDACTED
kind: Secret
metadata:
name: hvs-service-principal
namespace: argocd
type: Opaque
The Vault Secrets Operator refers to the HCPAuth resource to authenticate to an HCP project with the read-only service principal you created in the argocd namespace:
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPAuth
metadata:
name: default
namespace: argocd
spec:
method: servicePrincipal
organizationID: HCP_ORG_ID
projectID: HCP_PROJECT_ID
servicePrincipal:
secretRef: hvs-service-principal
After deploying the HCPAuth resource to the cluster, you can now define a resource to synchronize secrets from HCP Vault Secrets to Kubernetes.
»Sync secrets from HCP Vault Secrets
Use the HCPVaultSecretsApp resource to define the secrets VSO synchronizes from Vault to Kubernetes. You can define a destination Kubernetes Secret for the credentials and each key in HCP Vault Secrets will map to a key in the secret. If the name of the HCP Vault Secrets key does not match the required Kubernetes secret key you need for a workload, you can configure transformations for each key in the secret. VSO will also refresh the secret on an interval defined in the refreshAfter attribute.
For example, Argo CD creates private repositories by scanning for Kubernetes Secrets with the argocd.argoproj.io/secret-type label. The HCPVaultSecretsApp resource for this tutorial’s GitHub repository includes that label in the destination. It also reads each secret from HCP Vault Secrets and maps it to the keys required by Argo CD, such as githubAppID and githubAppPrivateKey. The repository secret for Argo CD also requires the type key, which is set to git.
apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPVaultSecretsApp
metadata:
name: github-creds
namespace: argocd
spec:
appName: argocd
destination:
create: true
labels:
argocd.argoproj.io/secret-type: repo-creds
hvs: "true"
name: github-creds
overwrite: false
transformation:
templates:
githubAppID:
name: githubAppID
text: '{{- get .Secrets "githubAppID" -}}'
githubAppInstallationID:
name: githubAppInstallationID
text: '{{- get .Secrets "githubAppInstallationID" -}}'
githubAppPrivateKey:
name: githubAppPrivateKey
text: '{{- get .Secrets "githubAppPrivateKey" -}}'
type:
name: type
text: git
url:
name: url
text: '{{- get .Secrets "url" -}}'
hcpAuthRef: default
refreshAfter: 1h
While the key names in HCP Vault Secrets do match the required keys for Argo CD, you add the transformations to demonstrate the value of re-mapping secrets and adding required fields, such as the type. In general, use the transformation field to create a Kubernetes Secret that conforms to the expected schema of any resource that uses it.
Once you apply the resource, VSO creates a Kubernetes Secret named github-creds with the fields and values defined in the transformation.
$ kubectl get secrets -n argocd github-creds -o yaml
apiVersion: v1
data:
_raw: REDACTED
githubAppID: ODU4OTMx
githubAppInstallationID: NDg2Mzg5OTI=
githubAppPrivateKey: REDACTED
type: Z2l0
url: REDACTED
kind: Secret
metadata:
labels:
app.kubernetes.io/component: secret-sync
app.kubernetes.io/managed-by: hashicorp-vso
app.kubernetes.io/name: vault-secrets-operator
argocd.argoproj.io/secret-type: repo-creds
hvs: "true"
name: github-creds
namespace: argocd
ownerReferences:
- apiVersion: secrets.hashicorp.com/v1beta1
kind: HCPVaultSecretsApp
name: github-creds
uid: 729d7860-0065-4802-b892-dffbe15bbffb
type: Opaque
Argo CD recognizes the secret because of the argocd.argoproj.io/secret-type: repo-creds label. It creates a repository resource linked to the repository URL and GitHub App.
Repositories in Argo CD UI for code
To verify any changes or review access to the secret, you can use the activity logs for the application in HCP Vault Secrets. The activity logs in HCP Vault Secrets indicate that the argocd service principal used by VSO has listed secrets under the argocd application.
Activity log in HCP Vault Secrets
If you create a new GitHub App and update the secrets in HCP Vault Secrets, VSO updates the github-creds secret with the new application IDs and private keys the next time it refreshes the secret. Argo CD updates the repository resource to use the new secrets without disrupting the repository connection. If you need to make changes to many GitHub Apps or credentials, you can update them all in HCP Vault Secrets without searching through Kubernetes namespaces and clusters.
»Learn more
If you currently store credentials in Kubernetes Secrets, you can copy them to HCP Vault Secrets and create resources to synchronize them into your Kubernetes cluster. This process avoids significant refactoring of Kubernetes workloads and lets you manage and track the secret lifecycles in a central location. While the example demonstrates how to synchronize secrets for Argo CD, you can use this pattern for other Kubernetes workloads and resources.
To get started, you can:
- Sign up for HCP to start using HCP Vault Secrets.
- Review our documentation to learn more about HCP Vault Secrets and check out our tutorials on using HCP Vault Secrets with Kubernetes.
- Find a complete list of available sources for secrets syncing in the Vault Secrets Operator documentation.
- Learn how to use Vault to manage API tokens for the Terraform Cloud Operator.
Sign up for the latest HashiCorp news
More blog posts like this one
Fix the developers vs. security conflict by shifting further left
Resolve the friction between dev and security teams with platform-led workflows that make cloud security seamless and scalable.
HashiCorp at AWS re:Invent: Your blueprint to cloud success
If you’re attending AWS re:Invent in Las Vegas, Dec. 2 - Dec. 6th, visit us for breakout sessions, expert talks, and product demos to learn how to take a unified approach to Infrastructure and Security Lifecycle Management.
Speed up app delivery with automated cancellation of plan-only Terraform runs
Automatic cancellation of plan-only runs allows customers to easily cancel any unfinished runs for outdated commits to speed up application delivery.