Vault Secrets Operator: A new method for Kubernetes integration
The Vault Secrets Operator implements a first-class Kubernetes Operator pattern for HashiCorp Vault along with a set of CRDs responsible for synchronizing Vault secrets to Kubernetes Secrets natively.
The beta version of the Vault Secrets Operator is now available as a final addition to the HashiCorp Vault 1.13 release. The Vault Secrets Operator is the newest method for Vault and Kubernetes integration, implementing a first-class Kubernetes Operator along with a set of custom resource definitions (CRDs) responsible for synchronizing Vault secrets to Kubernetes Secrets natively. This post will take a dive deep into this Operator and its place in the Vault ecosystem.
»Vault and Kubernetes: Better together
Kubernetes natively supports the management and distribution of secrets within a cluster. Applications running on Kubernetes can consume secrets natively from the Secrets API. Kubernetes Secrets can be consumed directly from files in a volume mounted by a container, through container environment variables, or directly from the API by an application that supports that retrieval mechanism.
In general, Kubernetes Secrets can be thought of as a static configuration object containing confidential information. Secrets are typically created during the deployment of an application and remain unchanged throughout the lifecycle of the application’s container. Kubernetes does not support dynamic secrets or rotation, leaving that responsibility to users.
Vault fills the gaps left in Kubernetes by providing a complete solution for secrets management. It consolidates identity brokering, secrets management (including dynamic secrets), rotation, and security policy compliance in one platform. By coupling Vault’s secret management feature set with a Kubernetes Secrets cache, the developer only has to consume the secret natively from Kubernetes.
»The Vault Secrets Operator
The current approaches to Vault/Kubernetes integration require some level of application modification if you want the ability to read from a file. Furthermore, applications need to be aware of when credentials have been modified so that they can be re-read from the file. In response to the user interest in a new, Operator-based method of Vault/Kubernetes integration, Vault 1.13 introduced the Vault Secrets Operator, which will allow users to natively sync secrets from Vault to Kubernetes clusters.
Kubernetes Operators allow the customization of automatable tasks within a Kubernetes cluster. This operator pattern removes some service management burdens from human operators, such as managing secrets for the services orchestrated by Kubernetes.
Today we’ve released the Vault Secrets Operator (beta), which implements a first-class Kubernetes Operator along with a set of CRDs responsible for synchronizing Vault secrets to Kubernetes Secrets natively. The Operator synchronizes static, dynamic, and PKI-based secrets from one or more Vault server instances. The Operator is also responsible for secret rotation and provides controllers for handling the various secret-specific custom resources (CRs) s. The Operator can also perform post-rotation actions, like notifying an application directly by triggering a rolling update of a deployment.
»How the Vault Secrets Operator works
The Vault Secrets Operator implements a controller that uses Kubernetes custom resources defined by the Vault Secrets Operator’s custom resource definitions to manage the secrets used by the services. The secrets are then managed by Vault, orchestrated in Kubernetes using custom resources, and consumed by the applications in a Kubernetes-native manner, cleanly separating concerns. Using standard Kubernetes declarative patterns, the Vault Secrets Operator reconciles the current state to the desired state specified in the CRs.
To check out the code, visit the Kubernetes Vault Secrets Operator GitHub repo. The project SDK is based on the Operator SDK for Go and provides the tools necessary for building, testing, and deploying a Kubernetes Operator.
»Authentication to Vault
During the beta period, we are supporting only the Kubernetes authentication method. This lets us focus on the core aspects around secret synchronization. Over time, we anticipate adding support for additional authentication methods.
Vault authentication configuration can be applied from the VaultAuth CR. The Secret CR includes a reference to a VaultAuth. The VaultAuth resource provides the parameters necessary to authenticate to Vault and perform the sync.
vaultauth.yaml
apiVersion: secrets.hashicorp.com/v1alpha1
kind: VaultAuth
metadata:
labels:
app.kubernetes.io/name: vaultauth
app.kubernetes.io/instance: vaultauth-sample
app.kubernetes.io/part-of: vault-secrets-operator
app.kubernetes.io/managed-by: kustomize
app.kubernetes.io/created-by: vault-secrets-operator
name: vaultauth-sample
namespace: tenant-1
spec:
vaultConnectionRef: vaultconnection-sample
method: kubernetes
mount: kubernetes
kubernetes:
role: demo
serviceAccount: default
The necessary Vault login credentials come from a ServiceAccount configured in the CR’s namespace. The Vault Secrets Operator requires both read and create access to any ServiceAccount configured in the related VaultAuth CR. The create permission is required to create short-lived ServiceAccount tokens.
By separating the authentication configuration from the Secret CR, we are able to provide future support for other authentication methods, and potentially other architectures or deployment strategies.
»Authentication policy controls
In upcoming versions, we plan to give the Vault Secrets Operator the ability to limit the supported auth methods via the command line. The code will resemble this snippet:
allowed-auth-methods=aws,gcp,token
These options add an extra layer of security by preventing unsupported and unauthorized auth methods from being used.
The Vault Secrets Operator will be able to constrain sync activities to specific Kubernetes namespaces. By default, all Kubernetes namespaces are allowed. The Vault Secrets Operator beta currently supports only the Kubernetes auth method, but more auth methods are planned for the GA release.
»Required permissions
The Vault Secrets Operator requires various Kubernetes permissions to function correctly. These include:
| Object | Permission | Reason |
| Secret | create, read, update, delete, watch | Sync operations, Vault auth |
| ServiceAccount | read, create | Vault auth |
| Deployment | read, update, watch | Post secret rotation actions |
If granting these permissions for the entire cluster is problematic for your team from a security management or compliance standpoint, consider restricting the Vault Secrets Operator’s access to objects in specific namespaces.
»Telemetry and operational reporting
The Vault Secrets Operator provides a Prometheus metrics endpoint, along with support for the Prometheus Operator. This allows users to be alerted to any potential issues with the system. The Prometheus output can be integrated with Grafana during the beta period. However, we anticipate providing a Visual Studio Online-specific Grafana dashboard for the GA release.
The Vault Secrets Operator gives users meaningful signals for determining the overall health of its secrets management processes, as well as analytics to measure security policy adoption.
Vault Secrets Operator metrics include:
- Number of managed secrets by namespace
- Vault request/response timings
- Number of secrets by Vault Secret type
- Number of Secret custom resources
- Number of Vault connections
- Number of VaultAuth custom resources
- Number of Vault authentication methods in use
- Kubernetes namespaces being serviced
- Sync operations
- Operator version and build information
- Sync failures
- Vault authentication failures
- Orphaned VaultAuth CRs from its VaultConnection
- Kubernetes API failures
- Orphaned Secret custom resources from its VaultAuth
- Duplicate Secret status
»Related Vault and Kubernetes integration utilities
To learn more about other Vault/Kubernetes integration methods, read our blog post on Kubernetes Vault Integration via Sidecar Agent Injector vs. CSI Provider and visit each method’s GitHub repository:
- vault-k8s: Provides access to Vault secrets by deploying a vault-agent sidecar into a Kubernetes Pod.
- vault-csi-provider: Fetches secrets stored in Vault and uses the Secrets Store Container Storage Interface (CSI) driver interface to mount them into Kubernetes Pods.
»Beta feedback
As you pursue your Kubernetes secrets management journey with Vault Secrets Operator, we’re interested in any feedback and recommendations you may have. To provide feedback on our beta, please log a GitHub Issue.
You can learn more about the Vault Secrets Operator by visiting the vault-secrets-operator GitHub repository. Further documentation on using Vault with Kubernetes can be found in our HashiCorp Developer guide to Kubernetes and Vault. To learn more about HashiCorp Vault please visit the Vault product page.
Sign up for the latest HashiCorp news
More blog posts like this one
Vault integrations with MongoDB, Private Machines, and walt.id strengthen customer security
Three new HashiCorp Vault ecosystem integrations extend security use cases for customers.
HashiCorp at re:Invent 2024: Security Lifecycle Management with AWS
A recap of HashiCorp security news and developments on AWS from the past year, for your security management playbook.
HCP Vault Dedicated adds secrets sync, cross-region DR, EST PKI, and more
The newest HCP Vault Dedicated 1.18 upgrade includes a range of new features that include expanding DR region coverage, syncing secrets across providers, and adding PKI EST among other key features.