Vault Transform: Protecting Secrets in External Systems

A common request we’ve had with HashiCorp Vault Enterprise is to protect application secrets stored in external untrusted or semi-trusted systems. Balancing the high availability of secrets in these systems, and protecting these secrets pursuant to strict compliance requirements such as PCI-DSS and HIPAA, is a challenge that the Vault team has been investigating for over a year.

The result of these efforts is a new feature we have released in Vault 1.4 called Transform. Transform is a Secrets Engine that allows Vault to encode and decode sensitive values residing in external systems such as databases or file systems. This capability allows Vault to ensure that when an encoded secret’s residence system is compromised, such as when a database is breached and its data is exfiltrated, that those encoded secrets remain uncompromised even when held by an adversary.

»How Transformations Work

There are two types of transformations available. First, Masking is a one-way transformation that allows for Vault to anonymize data per a custom character mask: for example the stars or ampersands displayed on an ATM screen when you type in a debit card’s PIN number. A key benefit with data masking is that your Vault server allows you to quickly redact or sanitize sensitive data based on criteria you define.

Second, there is a two-way transformation that allows for data type protection of the input structure, while maintaining that same data type of the encoded ciphertext. In the example below, you can see the example credit card number as the input, along with the encoded ciphertext following a similar structure.

Retaining a similar data structure for sensitive data that has been encoded gives operational flexibility in where you store it. For example, you might have a large customer database with billing details, where you would like to protect that data. By having the encoded ciphertext retain a similar format, you can now store that ciphertext back into your source systems with minimal database changes (no need to alter tables, etc).

»How Is It Different?

When protecting secrets such as credit card numbers and other PII data in untrusted or semi-trusted systems pursuant to compliance requirements such as PCI-DSS and HIPAA, companies typically protect data through the use of tokenization. Tokenization is a process that protects secrets by generating random token identifiers for data under its protection, brokering access to that data for trusted applications via a table or ledger associating tokens with their secrets.

Quality tokenization, combined with a strong random number generator, ensures that protected secrets remain secure wherever they reside. However, the format of protected secrets is not retained with traditional tokenization methods utilizing a lookup table or ledger, making it very hard, sometimes impossible, to manage tokenized data within an application or in storage itself. This presents often insurmountable challenges to implementation of a tokenization initiative.

Unlike traditional tokenization, Vault Transform procedure generates ciphertext that protects the data’s structure and format while maintaining the encoded values’ security. With full support for regular expressions, users can customize a pattern (e.g.: the XXX-XX-XXX of US social security numbers, where X is a 1-9 numeric value) and enforce the encoded value subscribing to the schema of that pattern.

Vault is able to ensure these encoded ciphertexts are secure thanks to its use of AES FF3-1, a revision to the FF3 algorithm that proves the algorithm’s resilience against chosen-ciphertext attacks and external attacks from future supercomputers while retaining the algorithm’s speed and performance. AES FF3-1 is a NIST-approved algorithm recommended in the upcoming NIST SP800-38G standard.

While processing these transformations, Transform does not actually store the protected secret, rather it protects only the key material necessary to decrypt the secret’s ciphertext. This maximizes the encode and decode performance for applications, while also minimizing the possibility of exposure of that secret.

»Why It Matters

Transform is a different form of encryption that focuses more on user-defined encryption parameters and structures so that organizations can protect highly sensitive information while retaining the structure and mapping the output. This empowers developers, operations, and security staff to securely work within their own workflows, data parsing, and discoverability while maintaining compliance and governance, all within a single endpoint.

Transformations in Transform can also leverage the full suite of platform capabilities in Vault Enterprise. They can be replicated and made highly available across regions and continents. They can utilize features such as performance standby nodes to ensure high performance even into the hundreds of thousands of requests per second (RPS). They can be segmented in namespaces for secure multi-tenancy, and access can be gated by both ACL policies and Sentinel policies.

Transform can even use entropy augmentation to securely leverage external systems, including hardware-based true random number generators, in ensuring very high cryptographic security in generating encoded values. This helps ensure the encoded ciphertext preserves forward secrecy of its protected data even in the event of future advances in classical and quantum computing cryptanalytical techniques.

»Next Steps

Transform is a feature in Vault Enterprise’s Advanced Data Protection (ADP) Module. For more information on the Transform Secrets Engine, see our Learn Guide or the documentation.

Also, if you enjoy playing around with this type of stuff, maybe you’d be interested in working at HashiCorp too since we’re hiring!