It is no longer a secret for anyone, security is a major issue for all companies and of course the management of TLS certificates is one of these issues.
However, certificate requests are rarely automated and still require contacting the team in charge of the PKI to generate them. Usually, this means the certificate is issued with a very long validity period as you don’t want to ask for a new certificate every day.
The problem is that this is no longer sustainable in a Cloud environment which is by definition extremely volatile and with a zero-trust network, even more for containerized environments.
This is why we often use certificates with a wildcard that we install on our Kubernetes platform, but that does not solve the problem, it only hides it.
In this article, we will see how to automate the creation and management of the lifecycle of TLS certificates in a Kubernetes environment with HashiCorp Vault and its PKI secret engine as well as JetStack cert-manager.
For the sake of simplicity and to make it repeatable for demos, almost everything is automated through HashiCorp Terraform, whether it be the deployment and configuration of JetStack Cert-Manager (using helm and Kubernetes provider) but also HashiCorp Vault (using Vault provider).
As explained is the introduction, we want to build a common and automated workflow to allow creation and lifecycle management of TLS Certificates in a Kubernetes environment for all applications that are exposed to the outside world.
To do so, on one hand, we will deploy a Vault Server which will act as Root PKI and Intermediate PKI and will provide API endpoint for issuing Certificates; on the other hand, we will deploy and configure JetStack Cert-Manager which is a Certificate Controller Manager and it will be integrated with Vault.
In terms of workflow, it can be described as follows:
Here’s what you need as prerequisites on your laptop (even if this demo could be done on any managed K8s cluster).
Note: For this demo, I will use a Java Application built by my dear friend Laurent Broudoux named Fruits-Catalog that I have forked for my own use.
First, even if it’s quite obvious, you need to clone the repos needed by this demo.
git clone https://github.com/nehrman/medium-kubernetes-pkiaas
git clone https://github.com/nehrman/secured-fruits-catalog-k8s
Note: This is optional as you can do it on your existing Kubernetes environment.
You need to install and configure Minikube on your laptop.
brew cask install minikube
minikube addons enable ingress
minikube start — vm-driver vmware — memory 8096 — cpus 4 — disk-size 50GB
You should end up with something like this:
brew install helm
In order to use the PKI Secret engine from HashiCorp Vault, you need to start a Vault server on your local machine.
curl https://releases.hashicorp.com/vault/1.3.4/vault_1.3.4_darwin_amd64.zip -o vault.zip
unzip vault.zip
sudo mv vault /usr/local/bin
Note: HashiCorp Vault server in Dev mode is not for production but for development only.
vault server -dev -dev-listen-address="0.0.0.0:8200" -dev-root-token-id="root"
Now, all prerequisites are met and we can move forward …
Ok, now, we’re ready to use Terraform and our code to deploy and configure everything we need to test and validate the automation of our certificate management.
Note: This article will not cover how to download and configure Terraform, as we supposed you already have it.
But first things first, even if everything is fully automated, it still worth it to understand what the code will do.
Don’t worry, it is not as complicated as it looks.
Now, let’s move on and configure our variables and run terraform to apply it.
Ok, let’s start with the deployment and configuration of all the components we need for our demo.
To be able to connect to Vault, we need to set up VAULT_TOKEN with this command:
export VAULT_TOKEN=root
That’s the only environment variable we have to configure for Vault’s provider for Terraform.
If you want to customize the domain name or any other parameters, all others variables are defined in the variables.tf file and can be changed as needed.
Note: Don’t forget to configure the Vault’s address in your variable file or Terraform will not be able to contact Vault Server.
That’s not the final step, but at least, you don’t have to do everything manually.
terraform init
terraform plan
terraform apply
You should end up with something like this:
Note: Store the Root CA somewhere as you’ll need it later on for your web browser.
You can also check the state of all the newly created Kubernetes objects by running these commands:
kubectl get sa -n default
NAME SECRETS AGE
default 1 134m
vault-sa 1 10m
kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-75f6cdcb64-9jcbb 1/1 Running 0 8m25s
cert-manager-cainject... 1/1 Running 0 8m25s
cert-manager-webhook-... 1/1 Running 0 8m25s
kubectl get sa -n fruits-catalog
NAME SECRETS AGE
cert-manager-sa 1 12m
default 1 12m
kubectl get issuer -n fruits-catalog
NAME READY AGE
vault-issuer True 11m
kubectl get certificate -n fruits-catalog
NAME READY SECRET AGE
fruits-certificate True fruits-certificate 13m
Then, you can check with more details the vault-issuer and fruits-catalog certificate with these commands :
kubectl describe issuer vault-issuer -n fruits-catalogs
kubectl describe certificate fruits-catalogue -n fruits-catalogs
We are almost done as everything is configured but we still don’t have an Application to show something nice and to test our TLS Certificate.
As we are almost done, let’s do the last steps in order to have a fully configured Application with a TLS Ingress route configured and validated by our PKIaaS from HashiCorp Vault.
As we use minikube and our local Docker environment, we have to configure our bash environment for using the docker daemon inside minikube:
eval $(minikube docker-env)
It’s time to deploy the app and make sure everything works as expected.
mvn fabric8:deploy -Pkubernetes -Dfabric8.namespace=fruits-catalog
You should end up with something like this:
Check also that you have a new pod in the fruits-catalog namespace :
kubectl get pods -n fruits-catalog
NAME READY STATUS RESTARTS AGE
fruits-catalog-6b45f4554d-k6wkp 1/1 Running 0 74s
mongodb-58845f7854-f2rpx 1/1 Running 0 40m
Now, let’s retrieve the CA Certificate Chain and configure our laptop to trust the certificate and test https connection to the app.
As we use Vault as Root and Intermediate PKI, you have to add the CA Chain to your favorite Web Browser to allow it to trust Vault.
Of course, in a real-life scenario, the ROOT PKI of the company will be used to sign Vault as an intermediate authority.
Tip: If it’s not already done, create an entry in your host with the minikube’s IP pointing to fruits.testlab.local
Look at the certificate and ensure it is correctly issued by HashiCorp Vault and your own PKI that we’ve configured in the previous steps:
Let’s take a closer at the certificate:
Have you seen that the certificate is only valid for 10 minutes? But wait, your application is still working with a valid certificate.
Yes, you’re right, HashiCorp Vault and JetStack Cert-manager can issue certificates but they will also manage their lifecycle. This means Cert-Manager is aware of the lifetime of the certificate delivered by Vault and it will request another one (before the end of life of the first one) automatically to ensure continuous operation of the Application while providing better security with short-lived certificates.
In this article, we tackle a challenge that most companies are facing and we demonstrate how to make the PKI as automated as possible. And, yes, we used Kubernetes to demonstrate that, but we can do more by automatically delivering certificates to Load Balancers, Web Servers (VMs) or anything else that needs a TLS certificate with exactly the same workflow.
We can go even further with HashiCorp Vault Enterprise by providing, for instance, namespaces where you can provide multi-tenancy to your internal or external customers.
Thank you for your time reading this article, it’s not the easiest subject but it gives you a glance at the power of HashiCorp tools coupled with JetStack Cert-Manager for managing TLS Certificates.
References:
Golden patterns for infrastructure and security automation workflows lie at the core of The Infrastructure Cloud. Here’s how to implement them using HashiCorp Cloud Platform services.
Learn how to run a 3-node HashiCorp Vault cluster as a HashiCorp Nomad Job and automate the cluster initialization.
Learn how to deploy the underlying HashiCorp Nomad infrastructure and configuration to run HashiCorp Vault as a Nomad job.