A new prescription for secrets management

Enabling the business of medicine

Since 1997, athenahealth has been supporting healthcare organizations through its many offerings ranging from electronic health records (EHR) and revenue cycle capabilities through population health, and many more. athenahealth supports more than 10,000 customers of all shapes and sizes, including over 160,000 providers. These customers naturally expect their private information to be kept safe and secure, as outlined in athenahealth's strict privacy policy. However, managing large volumes of sensitive patient records is not without challenges.

"For years, we'd managed secrets and data security in a proprietary secrets management solution and several purpose-built systems for specialized teams," says Jeff Byrnes, lead site reliability engineer at athenahealth. "This made it much more difficult to ensure we continued to meet the increasing security needs of our customers given the sensitive nature of the data athenahealth receives. We needed to standardize how we manage secrets to ensure that we can easily and continuously deliver high quality security services to our customers."

Lack of transparency required X-ray vision

As athenahealth's customer roster has grown, so have the number of patient and operational records the company is responsible for securely storing. With more than 40% of U.S. patients' records to protect, it is especially important for athenahealth to remain nimble with its security capabilities.

"A long time ago, we would run hundreds of bare metal servers with VPNs for secure access that we could manually manage ourselves and store secrets how and where we wanted," says Ganapathysaran Nambirajan, senior engineering manager, platform services at athenahealth. "But when that environment grew to tens of thousands of servers, we needed to evolve how we managed secrets and several different solutions by different teams started to appear. The burden of managing multiple home-grown systems and ensuring all of them met all compliance and security requirements became difficult and overly resource intensive over time for a small team to manage."

"The fact that everything was so siloed made systems access and governance much more complicated than it needed to be," says Nambirajan. "Any time one team needed access to systems or records managed by another team, they'd have to request a ticket and wait two or three days for them to resolve it. It was really difficult to track the volume and statuses of those tickets for real-time updates."

The sheer volume of tickets and disjointed processes for handling and logging them also ran up the cost of the resources needed to support various departmental needs, which elevated the need for athenahealth to look for a viable alternative.

"Our central support team usually acts as the last resort when other teams are having problems resolving an issue, often because of a lack of access or visibility," Byrnes says. "The number of interruptions during the work day and after-hours requests from our on-call resources continued to increase as the volume of secrets-related problems grew. Eventually we reached an inflection point and wanted to find a unified method for secrets management with self-service capabilities that could save us time, money, and a ton of worry."

athenahealth is using HashiCorp Vault to automate secrets management to securely store the secrets that protect hundreds of millions of patient and business records, improve service availability, and reduce human resources consumption.