Refresh Secrets for Kubernetes Applications with Vault Agent
Learn the system signal and live reload methods for updating Kubernetes applications when secrets change. See an example via a Spring Boot application.
Some applications include a HashiCorp Vault client library in their code to retrieve and refresh secrets directly from the Vault API. But what happens if you cannot or do not want to include a library in your application? You might run different types of applications on a Kubernetes cluster and want to standardize how they refresh secrets without significantly refactoring application code. Rather than write code to retry and refresh secrets from the Vault API, you can instead run Vault Agent as a sidecar, which reduces the need for your application to directly connect to Vault.
This post shows you how to use Vault Agent to update secrets for an application on Kubernetes using a termination signal or live reload when a secret changes. By relying on Vault Agent pushing a command to reload, your application stays updated with new secrets regardless of the programming language or application framework. Without Vault Agent, you would have to refactor your application to pull new secrets from Vault with retry handling and connection reloading.
»Demo Application with Vault Agent
This post uses a demo application named “payments-app” that submits payments to a processor and records the payment in a database. The application requires a database username and password generated by the PostgreSQL secrets engine for Vault. The “payments-app” uses Spring Boot, a Java-based framework. However, you can apply the patterns in this post to other application frameworks in different programming languages.
The database username and password for the application has a maximum lease of four minutes for demonstration purposes. Vault Agent renews the lease for the username and password before they expire. For the “payments-app”, add Kubernetes annotations to inject Vault Agent as a sidecar. Vault Agent runs as another container and handles the retrieval of new database credentials.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
replicas: 1
selector:
matchLabels:
service: payments-app
app: payments-app
template:
metadata:
labels:
service: payments-app
app: payments-app
annotations:
vault.hashicorp.com/agent-inject: "true"
vault.hashicorp.com/role: "payments-app"
## omitted for clarity
Next, configure Vault Agent to cache a Vault token and secrets after initialization. Enabling the cache ensures that the sidecar contains the secret and Vault token and avoids reloading the application when it starts.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
## omitted for clarity
annotations:
## omitted for clarity
vault.hashicorp.com/agent-cache-enable: "true"
## omitted for clarity
Finally, define a set of annotations for Vault Agent to create a file with the database credentials. Vault Agent retrieves the username and password and writes them to a file named /vault/secrets/database.properties in a shared volume mount. The suffix of the annotation should match the filename you want to create with the secrets.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
annotations:
## omitted for clarity
vault.hashicorp.com/agent-inject-secret-database.properties: "payments/database/creds/payments-app"
vault.hashicorp.com/agent-inject-template-database.properties: |
spring.datasource.url=jdbc:postgresql://payments-database:5432/payments
{{- with secret "payments/database/creds/payments-app" }}
spring.datasource.username={{ .Data.username }}
spring.datasource.password={{ .Data.password }}
{{- end }}
If you have secrets at different Vault paths, you can add different annotations for each set of secrets. The suffix of each annotation should match the filename containing the secrets. For example, the payment processor’s credentials exist at a different Vault path, so you would add a second set of annotations to write the API username and password to /vault/secrets/processor.properties.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
annotations:
## omitted for clarity
vault.hashicorp.com/agent-inject-secret-database.properties: "payments/database/creds/payments-app"
vault.hashicorp.com/agent-inject-template-database.properties: |
spring.datasource.url=jdbc:postgresql://payments-database:5432/payments
{{- with secret "payments/database/creds/payments-app" }}
spring.datasource.username={{ .Data.username }}
spring.datasource.password={{ .Data.password }}
{{- end }}
## omitted for clarity
vault.hashicorp.com/agent-inject-secret-processor.properties: "payments/secrets/data/processor"
vault.hashicorp.com/agent-inject-template-processor.properties: |
payment.processor.url=http://payments-processor:8080
{{- with secret "payments/secrets/processor" }}
payment.processor.username={{ .Data.data.username }}
payment.processor.password={{ .Data.data.password }}
{{- end }}
What happens when the database username and password change? Your application needs to reconnect to the database with the new username and password. As a result, Vault Agent needs to send a signal to your application to reload the new credentials.
»Update Secrets by Application Restart
One technique to update secrets for an application involves restarting the application instance with a termination signal like SIGTERM. When your application gets a SIGTERM, it handles the signal by disconnecting connections to external services and shuts down. Then, Kubernetes schedules a new application container. When you use this approach, make sure that you have multiple instances of your application available as the signal will shut down the application instance and disrupt your service.
You can configure Vault Agent to send a SIGTERM. To start, make sure your application responds to the SIGTERM by shutting down connections gracefully. For example, “payments-app” includes a Spring Boot property to gracefully shut down its web server after the application completes active requests. Set server.shutdown=graceful in the application’s application.properties file.
server.shutdown=graceful
Your application may require additional code to handle the signal and complete active requests.
To tell Vault Agent to issue the signal, add a SecurityContext for the “payments-app” to run as a specific user ID. Then set the vault.hashicorp.com/agent-run-as-same-user annotation to true in order to allow the Vault Agent to use the same user ID as the application. Vault Agent also needs access to the application container’s processes in order to issue a termination signal for reloading. Running as the application’s user ID and setting the shareProcessNamespace attribute allows Vault Agent to send the signal to the application process. Note that sharing process namespaces gives all containers in the pod visibility into each other’s processes and filesystems.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
## omitted for clarity
annotations:
## omitted for clarity
vault.hashicorp.com/agent-run-as-same-user: "true"
spec:
shareProcessNamespace: true
containers:
- name: payments-app
## omitted for clarity
securityContext:
runAsUser: 1000
runAsGroup: 3000
Besides sharing process namespaces, you need to define the command Vault Agent sends to the application. Add the vault.hashicorp.com/agent-inject-command annotation with the suffix for the database.properties secret to the deployment and send the SIGTERM to the process ID of “payments-app”, which runs as a Java process.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
## omitted for clarity
annotations:
## omitted for clarity
vault.hashicorp.com/agent-inject-command-database.properties: |
kill -TERM $(pidof java)
## omitted for clarity
When Vault rotates the secret, the Vault Agent injector retrieves a new set of database credentials, writes them to the shared volume mount, and sends the termination signal to the application. The application container shuts down and gets rescheduled by Kubernetes. When a new application instance starts, it reconnects to the database with the updated credentials.
»Refresh Secrets through Live Reload
Although you can use a termination signal across many application frameworks, you may find it disruptive to restart the application container each time a secret changes. As a less disruptive alternative, some frameworks support the ability to reload an application upon configuration changes without shutting down the application. Other application frameworks support automatic live reload when properties change. If you use live reload, you do not have to wait for Kubernetes to reschedule the application container.
For example, Spring Boot offers an actuator module that supports operations use cases like observability. Invoking its refresh endpoint triggers an event that signals to any interested components that something in the application has changed and that they should reconfigure themselves accordingly. The examples in the repository cover the additional dependencies and code for a Spring Boot application to live reload.
»Configure Application
Install org.springframework.boot:spring-boot-starter-actuator in your dependencies. Update the application’s application.properties to import configuration from files created by the Vault Agent injector using the spring.config.import property.
Finally, enable the actuator endpoint for /refresh.
spring.config.import=file:${CONFIG_HOME:/vault/secrets}/database.properties,file:${CONFIG_HOME:/vault/secrets}/processor.properties
management.endpoints.web.exposure.include=refresh
Note the spring.config.import property references the CONFIG_HOME environment variable. You can use this environment variable to override the shared volume path to the Vault secrets. If you do not define this environment variable, Spring reads the configuration from the default volume path at /vault/secrets.
When the application starts, it gets the database credentials from /vault/secrets/database.properties and connects to the database. Each time you make an empty HTTP POST request to the /actuator/refresh endpoint, the actuator checks for configuration differences between the application’s environment and the file properties. If it finds a difference, Spring publishes a refresh event.
You can have Spring recreate a whole bean with the @RefreshScope annotation annotation. Place this Java annotation on any Spring component or bean provider method to create a proxy object. The proxy object listens for refresh events and recreates the internal instance.
Note that @RefreshScope will completely destroy and then recreate a given bean with no regard to its internal state. For additional fine-grained control over an object, use a bean of type ApplicationListener<RefreshScopeRefreshedEvent>. Interested components may pull down the updated configuration and manually refresh their own internal state.
Define a database connection DataSource that refreshes each time the properties used to configure it have changed. Inject Spring Boot’s DataSourceProperties instance and create a refreshable DataSource bean.
package com.hashicorpdevadvocates.paymentsapp;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.context.annotation.Bean;
import javax.sql.DataSource;
@SpringBootApplication
// omitted for clarity
public class PaymentsApplication {
public static void main(String[] args) {
SpringApplication.run(PaymentsApplication.class, args);
}
@Bean
@RefreshScope
DataSource dataSource(DataSourceProperties properties) {
return DataSourceBuilder.create()
.url(properties.getUrl())
.username(properties.getUsername())
.password(properties.getPassword())
.build();
}
// omitted for clarity
}
For other properties, like the payment processor’s credentials, you can use @ConfigurationProperties or @Value annotations to define custom properties and refresh them.
package com.hashicorpdevadvocates.paymentsapp;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.cloud.context.config.annotation.RefreshScope;
@RefreshScope
@ConfigurationProperties(prefix = "payment")
@Data
class PaymentAppProperties {
private Processor processor = new Processor();
@Data
static class Processor {
private String url;
private String username;
private String password;
}
}
Inject the PaymentAppProperties instance and create a refreshable PaymentProcessorClient bean, which the application uses to submit payments to the processor.
package com.hashicorpdevadvocates.paymentsapp;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.context.properties.EnableConfigurationProperties;
import org.springframework.boot.jdbc.DataSourceBuilder;
import org.springframework.cloud.context.config.annotation.RefreshScope;
import org.springframework.context.annotation.Bean;
import javax.sql.DataSource;
@SpringBootApplication
@EnableConfigurationProperties(PaymentAppProperties.class)
public class PaymentsApplication {
public static void main(String[] args) {
SpringApplication.run(PaymentsApplication.class, args);
}
// omitted for clarity
@Bean
@RefreshScope
PaymentProcessorClient paymentProcessorClient(PaymentAppProperties properties) {
return new PaymentProcessorClient(
properties.getProcessor().getUrl(),
properties.getProcessor().getUsername(),
properties.getProcessor().getPassword());
}
// omitted for clarity
}
Upon refresh, Spring Boot reloads the new values in the Spring Environment object, which sources configuration keys and values from the application’s external property files. Vault Agent can now use the refresh mechanism to reload objects and values in the application.
»Configure Vault Agent
To configure Vault Agent to refresh configuration using the Spring Boot actuator, add the vault.hashicorp.com/agent-inject-command annotation with the suffix for the database.properties secret to the deployment and include a command to send an HTTP POST request to the application’s /actuator/refresh endpoint.
apiVersion: apps/v1
kind: Deployment
metadata:
name: payments-app
spec:
## omitted for clarity
template:
metadata:
## omitted for clarity
annotations:
## omitted for clarity
vault.hashicorp.com/agent-inject-command-database.properties: |
wget -qO- --header='Content-Type:application/json' --post-data='{}' http://127.0.0.1:8081/actuator/refresh
Vault Agent sends the refresh request to Spring Boot, which live reloads the database configuration without disrupting the web server. The live reload gracefully disconnects database sessions, reads in new configuration, and creates new database sessions while maintaining the application’s availability.
Note that the Spring Boot refresh endpoint could trigger disruptive changes to your application, such as creating new data sources. Secure individual actuator endpoints by restricting access, making them inaccessible to public traffic, and adding certificates.
»Conclusion
You can use Vault Agent to standardize the retrieval and refresh of secrets across different application frameworks running on Kubernetes. For applications that support a live reload, add a Vault Agent annotation to the deployment to make a refresh request to the application. If your applications do not support a live reload, use termination signals to restart the application.
For more detailed configuration information, review the demo application’s code repository. Use additional annotations to configure multiple secrets and settings for the Vault Agent sidecar injector.
If you use Spring and want to refresh configuration directly from secrets in Vault, check out the documentation for Vault as a backend for Spring Cloud Config Server. For Spring applications that need to access the Vault API directly, review the documentation for Spring Cloud Vault. The library supports the refreshing secrets and encrypting data using Vault’s Transit secrets engine.
Thank you to Josh Long, Spring Developer Advocate at Tanzu, a division of VMware, for educating and clarifying Spring Boot details for this post.
Questions about this post? Add them to the community forum!
Sign up for the latest HashiCorp news
More blog posts like this one
Fix the developers vs. security conflict by shifting further left
Resolve the friction between dev and security teams with platform-led workflows that make cloud security seamless and scalable.
HashiCorp at AWS re:Invent: Your blueprint to cloud success
If you’re attending AWS re:Invent in Las Vegas, Dec. 2 - Dec. 6th, visit us for breakout sessions, expert talks, and product demos to learn how to take a unified approach to Infrastructure and Security Lifecycle Management.
HCP Vault Secrets adds enterprise capabilities for auto-rotation, dynamic secrets, and more
HCP Vault Secrets focuses on making a fast and easy path for secure development with key new features including auto-rotation (GA), dynamic secrets (beta), a new secret sync destination, and more.