Vault Automation with Terraform and Packer at OneMount

Using HashiCorp Vault, VinID (a OneMount Company) reduced time spent on secrets management by 90%. Learn how they automated processes with Packer and Vault.

»Transcript

I will show you everything about security management at OneMount Group. We started OneMount  2 years ago. 

Our product VinID offers e-commerce financial service on mobile. VinID is the payment gateway. We also have VinShop and OneHousing for e-commerce in Vietnam.

Read the VinID and Vault Case Study

We are among the top 5 e-commerce systems in Vietnam. We provide digital financial services for every company in Vietnam. In the future we will have more products. At the moment, we have around 10 million customers.

My name is Quan Phuong-Cong and I am the head of the system platform team at OneMount Group. We provide the API gateway, the message queue system, the logs central system, and the secrets management system. 

We build everything in the cloud. At the moment, we use GCP. In the future we might also use Amazon Web Services or have the datacenter on-premises.

My team has only 10 people, but we provide everything for the share system platform. We use Vault and do automation with the HashiCorp stack, including Terraform and Packer. 

»Vault for Secrets Management

In OneMount Group, we have many problems with secrets management and certifications. We have PII and sensitive data because we work in financial services. So we need to encrypt everything and mask all data before we move it into the database, before we move it into the log centralization system, and before we do ETL.

We need secure communication because we have reviews of our microservice system and we need to secure every connection between services in our system. We use mTLS, SSL, and HTTPS, and everything depends on the certificates and the CIE and the public key infrastructure.

We needed to build the system to cover everything with encryption, install the secrets management and the permissions. We needed to build a system that can work for our many users in G Suite. We needed to adapt the application so the user in G Suite can log into the system and retrieve data like a secret or a certificate.

When we started the company 2 years ago, we found out about HashiCorp Vault and decided it was the solution to fix everything. Vault acts like the secrets management system. 

The problem was, How can we deploy Vault into GCP and into the on-premises environments because we need to count on the on-premises environment and the GCP environment for everything. 

To deploy to GCP, we used Vault Instance Groups with auto-scaling and auto-healing features. The idea was that we could push Vault, Packer, and Terraform into the system using Instance Groups and GitLab.

We started the Instance Groups with a small subnet. A friend asked me once about why we do everything with small subnets. It's because we need to pre-create all DNS records and import the DNS records into the compilation of the Vault server. Having so many IPs, we can create all the DNS records to the compilation of Vault. So we started with a small subnet and we do the vertical scale for Vault server.

After that we set up the Instance Groups with the backup scheduler with Packer for pre-built Vault images and we install Vault into the image in GCP and save it into the image and we configure the instance template with the Vault image. When we configured the VM, we used the start script. You can see this is my demo configuration when we do the start script for Vault into the Instance Group.

And we use GCP KMS to install the seal key, because when Vault starts we need to unseal the data that's encrypted in the Vault server and we install the key to unseal in GCP KMS. We use Raft storage for everything. We don't use Consul because we don't need to manage more clusters; we have Vault. And so we don't need to have Consul; for that we use Raft.

At the moment, we don't have any problems with Raft storage. We can do unseal, we can recover all data in Raft storage in the cluster. 

We use Terraform to manage the Vault resource. We manage the permission, we manage the PKI, we create the PKI, we generate the certificates via Terraform. We use the tools of HashiCorp to manage the secrets management system.

»Vault Integration

Because we do everything with the enabled GCP authentication in the Vault cluster and we have LDAP in Google, we enabled LDAP authentication in the Vault cluster, AppRole for authentication, and the token is enabled by default. We have 3 use cases with Vault. In the key-value store, we store the database, the encryption key, and much more. We have hundreds of applications running on hundreds of GKE clusters, and we have 1 cluster of Vault. We store the key-values in Vault. When applications start, they will retrieve the secret on key-value and apply it into the application running on GKE with port.

Sometimes we have some applications running on VMs, and we do the same thing. And we have an automation script running on cloud. Just like Google App Engine, we use secrets management in Vault and we store the secret's key-value and retrieve it in the Google App Engine. 

Who can import and export the secrets on key-value store? At the moment in OneMount Group, only a team leader or C-level can have write privileges in Vault. Applications have read-only permission.

»mTLS and Certificate Management

We use mTLS because we have around 1,000 applications. We encrypt everything in our system and so we need to generate the certificate and manage the certificate for the mTLS connection. We use Vault for that. 

We enabled GitOps. We have CI/CD for Terraform and we generate the certificate in the Terraform automation script. Only the SIE team can encrypt it and write. After that we store the certificate in the key-value store that the app has generated from the script in Vault and Vault can help us to manage the PKI.

PKI has a product key, and we import everything in Vault running on GCP. Vault can manage and encrypt our PKI with the certificate and product key and we don't need to do anything. We use both the key, the site certificate, and the generated certificate because Vault can provide us 2 official certificates, the signing certificate and the generated certificate, and we use both.

»Integration with Apps Running on GKE

In this slide I will show you how we can integrate with the application running in GKE. We deploy everything into the port in GKE, and around 90% of our applications are running on GKE. So we need to integrate everything into GKE like authentication and authorization with Vault and with GKE. We use G Suite authentication, we enable the cloud IAM, and we generate the signed JWT for the client or, with applications running on GKE, we do it automatically. 

We do it with the service account. We generate the service account and sign the permission to the service account and deploy the service account into the port. After that, the port will start and GKE can have a service account, can sign the JWT, and can connect with Vault with the JWT that's signed to the GCP. Vault server can verify the service account with the JWT to the cloud IAM and Vault can identify what applications accept Vault and what permissions the application can do with Vault. It is our logic when we built these applications on GKE with Vault.

On screen now is my separate code for the application. You can see we do everything in code. We sign the JWT with the function in automation, we create the permission and create the token, we do everything in Vault with Terraform, and we store everything for the configuration on GitLab. We use the CI/CD to deploy the application and Vault secrets automatically.

»Guarding PII and Sensitive Data

The last thing that I want to show you is about our PII and sensitive data. Because we are a financial service, our data can show the information of the customer like names, mobile phone, and maybe the credit card. We need to encrypt and mask everything in our database. We need to anonymize the data before we send it into the database, and we need to meet PCI-DSS compliance, plus many more regulatory regimes.

We do everything in our applications, and the masking and anonymizing depend on the application. We encrypt the data in applications, and we decrypt the data in applications when we retrieve the data from the database. We have many applications and many teams to write out the applications, so we have many solutions to encrypt the data in the applications.

This is our problem. Our expectation is we need 1 thing to cover the encryption and anonymization. Two years ago when we found out about Vault, we expected Vault to be able to encrypt everything for us, and Vault Assist is the feature that provides this. When we push into Vault Assist, the data will be masked and then encrypted and we can save it into the database. 

We need to test because we have so much traffic and we have so many applications, and sometimes we have a high load and we need to know how well the cluster can serve the data at the request to our Consul.

Some numbers can show we are completely applying Vault into our system. We have 3,000 certifications managed by Vault. We installed around 300 application secrets, running on around 100 GKE clusters in Vault, and we can serve hundreds of employees. Employees can log into Vault and retrieve data. They store the passwords in Vault and we sign the permission for the user. We have hundreds of users with hundreds of kinds of permissions. We have around 3,000 leased tokens active at the moment. That is around 3,000 certificates, 3,000 applications that are running on our system. So we have 3,000 leased tokens and we manage Vault with clusters that are running on GCP.