HashiCorp Vault helps New Relic manage secrets for their digital intelligence platform

This is a guest post by Matthew Lapworth, Senior Application Security Engineer at New Relic. New Relic is a leading digital intelligence company, delivering full-stack visibility and analytics with more than 14,000 paid business accounts. The New Relic Digital Intelligence Platform provides actionable insights to drive digital business results. Companies of all sizes trust New Relic to monitor application and infrastructure performance so they can quickly resolve issues, and improve digital customer experiences.

At New Relic, our systems and infrastructure had grown, and we were facing challenges with securely storing and managing credentials. HashiCorp Vault has provided us with a consistent approach to manage secrets and credentials.

Challenges with Secrets Management

Our platform had grown to the point where we integrated with numerous internal and third party systems. Each of our services needed dozens of credentials to function. Credential management became an obstacle, leading to credential storage practices that were inconsistent. The difficulty of managing credentials resulted in serious challenges for teams seeking to collaborate on a single code base.

Changing a credential used by a service involved coordinating with multiple teams, who each had a copy of the changed credential. When a security event occurred, changing one or more of these credentials was necessary. However, due to the infrequent nature of these changes, procedures were not well understood and led to frequent service outages caused by a team not being informed of the change or the inclusion of incomplete features and functionality in the change.

Securing and Managing Secrets for Our Product Suite with HashiCorp Vault

We utilize Vault to manage secrets and credentials and also two other HashiCorp tools for our complete secrets management solution. HashiCorp Consul stores and provides high availability for the secrets management system and HashiCorp Terraform controls the lifecycle of the secrets management cluster.

The initial use case of Vault is to house the numerous credentials used by the services that make up New Relic’s product suite. Each team is assigned a partition in the generic credential backend to store their service’s secrets. This approach allows the teams to decouple rotation of secrets from the deployment of their service and to store their secrets in a tightly access-controlled location.

Our architecture includes a Vault cluster for our integrations and another for our production environments, running in our production AWS account. This VPC is connected to our primary data center via Amazon Direct Connect. We have 5 hosts running Vault, with 5 hosts running Consul as the storage backend. We use a load balancer with health checks to point to the Vault master.

Throughout the process there were several things we learned that will help us going forward:

• Ensure you have a well thought-out and practiced process for on-boarding services. This is especially important when first rolling out Vault. When developers at New Relic heard of a secure and easy-to-use system that would solve their credential management issues, they were more than willing to do the work to migrate.
• To ensure a smooth rollout and no service interruptions, we had a few, low complexity services migrate to Vault initially. This allowed us to test out our on-boarding process and shake out any assumptions that weren’t already explicitly called out in our discussions and documentation.
• Before rolling out Vault, take special care to plan out how secrets will be partitioned per team and per service. This can be difficult to change once Vault is heavily used. An ounce of planning is worth a month of maintenance windows.

In the future, we plan to use Vault for continuous credential rotation utilizing the various dynamic secret backends to create short-lived access credentials. This would include issuing certificates from the built-in PKI to services for client authentication and service-to-service encryption in transit. Additionally, we’ll be leveraging the dynamic secret backends more to automate access and revocation to things like AWS, host access via a bastion+SSH, and certificate deployment using the built-in PKI.

Vault has become a core part of our security strategy at New Relic. It, along with a few other services, creates the foundation of infrastructure that abstracts complex topics like credential management, AuthN/AuthZ, and makes the secure thing to do also the easy thing to do.

HashiCorp Vault is a product to secure any infrastructure and any application. Vault provides a consistent workflow to securely store and manage secrets, provide encryption as a service, and granular privilege access management. To learn more about HashiCorp Vault visit https://www.hashicorp.com/products/vault/.