Taming Application Secrets at BPS with HashiCorp Vault

Journey through the long-standing challenge of dealing with secrets in application architectures and how the Italian bank Banca Popolare di Sondrio is tackling it.

Speaker: Diego Braga

»Transcript

Welcome to our speech on Taming Application Secrets in Banca Popolare di Sondrio. I am Diego Braga, and let me introduce who we are.

Banca Popolare di Sondrio is a retail bank based in northern Italy. It was founded in 1871. This year we make 150 years, so Happy Christmas. We have more than 500 branches distributed between northern Italy and Rome, and we have more than 300 employees.

Our IT described by the numbers is; we have two datacenters, more than 2,300 servers that are mostly virtualized after some replatforming. And if you are curious about the distribution of operating systems, we have 60% of Linux and 40% of Windows. If you are concerned about that, yes, we do have a mainframe. As a perimeter of applications, we have hundreds of intranet-facing services and applications — and dozens of internet-facing services and applications.

As you can understand, we have a gazillion user IDs and passwords. I should make a scary face in this moment — and this is the face of a gazillion user IDs.

»One Identity Manager To Rule Them All (Or Not?)

What should be the solution to rule them all or not to rule them all? The first item that you can think is an identity manager. But let's take some steps back. We are in the late 90s or early 2000’s, and we rely on our own mechanisms of authentication/authorization for services. 

There is no integration between platforms. There are no common policies for users, authentication, and passwords. There are some techniques that come to the rescue, like Identity Management, User Directory, or single sign-on. Introducing digital identities and the lifecycle management of them let us live better with our credentials handling.

With the introduction of these technologies, we got rid of local user stores — and this was working fine, but only for real users, like physical ones. The main problem is still with apps. We have a metadata middle issue, but for applications. 

We call it an app stack in the middle issue. I'm talking about one that’s based on application servers that are relying on configuration files based on plain text for connecting to databases — to queue managers — to any service that can answer our web service or REST API language, and legacy backends. 

This wide range of different application stacks wasn't standardized in the way we managed the application credentials. It was difficult to enforce policies. Today, there is password rotation, expiry tokens, but back in that time, there was no way to enforce these policies. 

»Live with it? 

How do we tackle this issue? Like the old way, it looks like it was a thousand years ago, but as technology moves fast, this was the old way. So, we would handle this approach with the manual rotation passwords. And I know that you already have some headaches. 

How can our manual rotation password can be addressed? With painful manual rotation, with risky operations that can create incidents, because we can sometimes make mistakes — we are humans. And a risky operation could create an incident, which for sure grow over time. This technical debt creates less confidence with handling password management. And, for sure, you don't rotate passwords anymore in the future because you don't want to create incidents.

What was another way to handle password rotation? With the password management for application credentials, to do periodic audits and brute force checks on credentials. This can lead to rotations, but it could be difficult if you do it too late — if the application is already written in a bad way.

A third way to live in the old way was fencing access to the environment. So, limit the number of users and operators on production environments. This leads to more difficult activities in cases of troubleshooting — that is critical for any of this data that can happen in production. As I said, all these periodic audits that we did, there were further audits on separate non-production environments, like in early shift-left approaches. But this was still a wrong way to handle passwords and application credentials. And so, this was the late 90s or 20s. 

»Fast Forward to Cloud Native Architectures

Let's do a fast forward to cloud native architectures. These cloud native architectures are leading the hype in these moments. What are the main advantages? Kubernetes was the disruptor of the cloud native architectures because it forced us to reason about decoupling — decoupling by default, decoupling by design. These are still decoupled, and still complicated architectures — but I don't want to talk about that because we need another conference.

Let's move back to secrets. Secrets in Kubernetes are treated as a first class item. They are not an afterthought. You need to work with secrets immediately. But this is nice — this forces us to work in a precise way. But, what is the drawback? It only works for cloud-native-ready-applications, and environments that are cloud native.

»Vault to the Rescue

Vault came to the rescue because Vault can be adopted easily for cloud native applications. There is a native Kubernetes integration, which we leveraged in the first phase. This was the quicker phase in which we immediately understood that Vault was the solution for our business problem — for our security concerns, about applications that are based on Kubernetes that needed to connect to external services with credentials. 

Vault was the answer to our issue. Vault came to the rescue for these new cloud native applications. But, we have some things that are not yet cloud native. So, we need to retrofit these best practices that we understood with the adoption of Kubernetes and Vault, for, let's say, common application stacks and backends. 

I can give you an example of what we faced within the company. J2EE application servers or Oracle databases or Postgres databases and Active Directory, Backend services. This was our second phase in which we handled the adoption of Vault. We are adopting stricter policies because now that we know what Vault can give us as an advantage, we can work with policies, with rotating the password in an automated way, without the manual operation that I was talking before. These allow us to really decouple the backend from the external services that it needs to contact.

»Further Value

We have a next phase — which we will start in a few weeks — in which we will gain further value from Vault. Vault can give us an increased visibility on what is happening within the app — that needs to connect to other services with credentials — so we can leverage logs and auditing. That is another strict requirement, — especially in the financial services area — in the banking sector. 

We no longer want to write policies on documents and best practices that stick to a file written somewhere — documented somehow. We want to translate them into Vault policies. Corporate security policies are going to be translated into Vault security policies. This will give us a stronger adherence to what our company enforces as best practice security policies. We are also going to add external compliance policies based on how we handle credentials within our company. 

Vault has answered a lot of requirements in our company — the gazillions of application credentials that we're retaining in configuration files, in plain text, and even in the source code — I shouldn't say that, but it happens.

Adopting Vault, we started in the first phase, from cloud native architecture, but that was our greenest field of our application perimeter. We adopted Vault as a second phase into the common backends. We started with application servers — and this is working well — so we leveraged the Vault agent as our technology for decoupling applications connecting to Vault. We also have a nice way to decouple how applications can retrieve credentials from Vault.

Now, that Vault is adopted within our company — and in our IT department — we are shifting the security policies left and working with logs for auditing, and policy refinements can be written from our security engineers and IT engineers, to adhere to stronger internal corporate compliance policies — and external ones, based on European banking regulations.

Thank you so much for listening to my talk and have fun during this conference, as much as I'm having. Ciao Tutti.