How ABN AMRO is implementing DevSecOps in a multi-cloud environment

ABN AMRO's Stefan Simenon and his development team are busy implementing:

• CI/CD
• DevOps
• Multi-cloud environments
• Infrastructure as code
• Containerization and container security
• Compliance as code
• Secrets management

This talk will cover lessons learned on this journey, both the technical and organizational. See how they are using HashiCorp Terraform and Vault to help them in this massive initiative.

Transcript

Thank you for the introduction. I'm going to elaborate about our DevSecOps implementation within ABN AMRO. I will elaborate upon our CI/CD journey, about our multi-cloud implementation and how we implement security in our pipelines.

Stefan Simenon—already more than 20 years working within ABN AMRO. Started as a programmer and for the last four years. I'm responsible for CI/CD, for tooling, for application security and software quality. Last year we started implementing a managed container platform within ABN AMRO.

ABN AMRO, one of the largest banks in Europe. Mostly retail. It's headquartered in this city—in Amsterdam—and we are an agile organization. We have around 400 plus agile teams right now.

Starting the CI/CD journey

In 2015 we started our CI/CD journey. We used to be a waterfall organization. We found out that we were slow. Our competitors were becoming faster.

We started a program to do a CI/CD implementation. That program consisted of two parts. One part was all about the technical implementation. Implementing the right tools, the right pipelines, the right integrations, the right infrastructure to make it scalable. The second piece was all about the changed way of working. Because if you do CI/CD—if you do DevOps—you need to change your whole way of working. You need to change your whole mindset, and then that piece we have all the focus.

We created our pipeline. We started with CI. We started in lower environments with the CI and frontend. Once we had mature CI pipelines, we started extending technologies, and we started implementing our CD pipelines. In 2017 we integrated security in our pipelines. I will tell you later how we did it and what the current status is.

We have a central IT authority organization. That means that the teams are autonomous. They have lots of freedom. But in order to standardize—to scale—their autonomy is bound within certain fences. There is a certain set of tools prescribed, and every team needs to use that.

In 2017 we also set up our hybrid cloud environment. We already had a private cloud, and next to our private cloud, we implemented an Azure and an AWS environment. After 2017 it became a more continuous improvement program. We started our container journey, and we increased focus via CI/CD metrics.

Midrange build and delivery pipeline

We use a lot of tools for agile management. We have a distinction between CI and CD. CI is managed by Jenkins. Our CD is managed by XebiaLabs Release. A deployable artifact is generated out of CI which we store in Nexus, and we have all kinds of quality checks in our CI pipelines.

We use SonarQube for code quality. We use Fortify for secure coding. We use Nexus Lifecycle for open-source library management. Once the checks are passed, an artifact is generated which will we be picked up by the CD pipelines. At ABN AMRO, we have currently loads of pipelines in place which enable CI/CD for the agile teams.

These are the pipelines we have within ABN AMRO: A central team is delivering the pipeline building blocks. We have an inner sourcing concept. All kinds of communities and developers contribute to the pipelines, but within ABN AMRO the standard pipeline building blocks need to be used as much as possible. Within ABN AMRO, 70-80% of standard pipelines are being used.

As you can see there, there are lots of different technologies. For most of the technologies, we have a CI/CD pipeline in place which enables teams to do automated deployments.

CI/CD awareness program

We paid lots of attention to educating our IT leads. To roll out CI/CD—to roll out DevOps—the IT leads need to know what it is and how it needs to be managed. We had lots of events around CI/CD, and we are paying lots of attention to our developers. For us, it's very important that the developers have a good working environment—that they feel happy. Because the happier they are, the more productive they are.

They have all kinds of communities set up. We organize lots of hackathons—lots of meetups. We have lots of facilities in place to share knowledge and to share failures so that people can learn from each other.

We focus a lot upon learning because the technology is evolving a lot. Something which is hype right now can be old-fashioned in half a year, and the skills we need are simply not available in the market.

We need to focus a lot on educating our community. We have a 70:20:10 methodology. This which means 70% of agile team time can be spent on business functionality, 20% on IT4IT improvement and each and every person within ABN AMRO should spend 10% of their time on learning—on getting better—following seminars like this, getting certificates, following webinars. So 10% of the time they should spend on learning.

Prerequisites for CI/CD and fast deployment

Once you have the pipeline in place—once you have the right culture in place—many people think that they are done, that they are ready with CI/CD. But then the real journey starts. In order to become fast—to do multiple deployments per day, per week, per month—you need good quality software.

Within ABN AMRO in the past—not only ABN AMRO but also other companies—lots of monoliths that have been programmed. If you want to become quick, these monoliths need to be decomposed. They need to be optimized, and lots of automation needs to be in place.

We need to do trunk-based development, and we need to do automated unit testing. Teams that want to be quick need to invest a lot of time in this journey.

Including security in a DevOps environment

We have multiple security implementations. I will elaborate on our secure coding and the security of our open-source libraries. We started an initiative called Infra as Code and Compliance as Code. Since we're doing lots of things with containers, we are also focusing on container security and secrets management.

Open-source software is being used more over time. The good aspect of open source is that lots of code can be reused and developers can become more productive. But the danger is if you don't use the latest version then you will have lots of vulnerable software in production. Many companies have suffered from hacks because of open-source vulnerabilities.

The most famous one is Equifax. You see that it has impacted 140 million people, but everywhere in the world, hacks are being done. Yesterday I saw a message from British Airways. 500,000 people got impacted by a hack, and because of the GDPR law, British Airways is going to be fined 200 million.

You see the hacks are having more impact. In the past, it took a hacker 40-50 days to prepare and to conduct a hack. These days it only takes them 3 days. More open source is being used, but the risks are also increasing, so we pay lots of attention to good open-source software.

I conducted a webinar, together with Sonartype, about the threat and how we manage it. If you are interested, you can follow this up on the webinar.

This applies for secure coding; this applies for secrets management—everywhere where you have vulnerabilities. You have a chance that will you get hacked—and it will have this kind of impact.

Improving security at ABN AMRO

Security training and meetups

If you talk about security, it's not only a technical thing. We have lots of focus on the knowledge of the development community. We took lots of training. We organized lots of meetups. Our SecOps department organizes a green light festival twice a year. It's an internal ABN AMRO security seminar with lots of security updates and lots of security knowledge sharing. This is all to make the people aware that security is important.

Security quality gates in pipelines

If there is poor code quality or if insecure software delivered, the Jenkins pipelines will fail. Developers are then forced to fix the issues before generating a deployable artifact.

We implemented this 2 years ago. We had lots of resistance in the past. Before we implemented it, we aligned with our senior management. Our senior management they backed us up. But over time, people started to understand that it only helped them to deliver more secure software. Going forward, this has helped us to not use vulnerable software.

Secure coding team

We have a secure coding team within ABN AMRO. This secure coding team is setting a standard for security. In case there are false positives in the organization, they can be contacted to do a final check. This team is also keeping the rule sets up to date.

This team is, on the one hand, increasing the barrier for the development community, but on the other hand, they are helping the agile teams to fix their secure coding issues. My advice is if you're going to do this, make sure you have a team like this in place.

Monthly security team rewards

In the past, we did business with application development vendors. If they delivered poor quality software or if they delivered software late, they had a chance of getting a penalty.

Now if you move to CI/CD—if you move to DevOps—then you need to create a culture in which failing is okay. In which people are learning from the failure, so instead of giving penalties, we award teams for doing the right things. This reward can be a session in front of the management team or dinner. Every month there is a security award given to a team that does a good security job.

Security dashboard for senior management We have a dashboard for senior management where we've scanned most of our code. Pipelines can prevent new issues in production, but we also have lots of legacy which is not being touched. We also scanned this legacy software, so our most vulnerable issues are visible. They are visible for the senior management, and they can steer based upon that to make sure critical issues are mitigated.

Up to date open-source policy We have an open-source policy that is updated every year.

The goal of all this is to create a mindset that everybody feels responsible for security. It's not one department or only senior management that is responsible for security. Everybody should feel responsible for improving security.

Today we use Nexus Lifecycle from Sonartype for open-source library management and Fortify for secure coding. It means issues like SQL injections and cross-site scripting can be prevented. We have increased security awareness a lot, and we reduced the amount of critical issues by more than 50%.

Very important—security is now taken into account as a factor for making decisions. We have seen applications with many vulnerabilities, and we have decided not to fix the issues. In some cases, applications are being re-architected or newly built to make sure that we have good quality software.

Moving towards a container-based approach

Last year we started our cloud native journey. More developers are using containers within ABN AMRO, and we need to have an environment on which containers can land. We are using containers, mostly because of improved time to market. You can make sure that the same piece of software is implemented in each environment. It's cost-efficient. Running a container is cheaper than running a VM. You also see is that more third-party suppliers are delivering their software as a Docker image, so more teams are using containers.

If you look at the cloud-native landscape, then you see lots of possibilities—lots of options. This sheet is from last year. If you take the latest CNCF sheet, you probably will see more tools. If you leave everything up to the teams, you will get a wild growth of tools and solutions.

This is not what we want within ABN AMRO. We want clear guidance. We want standardization. We need to scale solutions. We want to have the best cloud features—to be easily consumable—and we would like teams to share knowledge. The teams that have implemented something good educate other teams so that other teams can learn from it.

Introducing Stratus—ABM AMRO's cloud container platform team

We have set up a separate cloud container platform team named Stratus. Stratus—if you look it up on Wikipedia—is a set of low-level clouds characterized by horizontal layering with a uniform base.

This is a metaphor: Uniform base means standardization. We would like to standardize things. Clouds. That means hybrid clouds, private clouds, multi-clouds. There is a big dependency on clouds. The low level means very close to the working environment—very close to the development teams.

This team works very closely with the development teams within ABN AMRO. This team has a mission. IT needs to enable development teams to deliver secure software in a fast way, and ensure 4 important things:

• Easy to use: The team needs to create a platform on which development teams can deploy their software. Development teams—they only should worry about programing software, updating software and deploying it with one push on a button to a production environment.
• It needs to be secure: We are a bank. We are a highly regulated company. We need to facilitate security.
• We need to be portable: We have a multi-cloud environment. We have two public clouds, one private cloud. We need to be able to switch across environments over the 90 days. That's a rule from the regulators.
• The software—the platform needs to be reusable. Something which is being programmed for AWS should be reused for Azure or maybe for a Google platform if we might be using that in the future.

What does our container framework look like?

First, you have a platform. That's a collection of frameworks and tools that are connected to each other. I will show it later. You need to have good governance in place, so you have a container platform team. You have the DevOps teams. You have the AWS team. You have the Azure team. You need to be clear who's doing what.

There will be shared responsibilities across teams. It's very important that these shared responsibilities are clear and that teams work very closely with each other.

Once you have the platform—once you have the right governance in place, you need to have the pipelines in place. That's because every application landing on this hybrid cloud—on this container platform—that only can be done via the pipeline. Manual deployments are not allowed anymore. So, therefore, we have a set of standard pipelines in place. When this is in place, then teams, applications, can land on the platform.

What does our cloud platform look like?

We have Amazon cloud, and we have Azure cloud, we implemented it two years ago. We have around 100-150 workloads running on Azure and Amazon. We are going to use Terraform as a cloud-agnostic tool for infra provisioning. We already had Nexus in place and Nexus Repository. We're going to reuse it.

For persistent storage, we need to find a solution. For container runtime and network, we use existing standards. For orchestration, we have chosen Amazon EKS, and Azure—AKS.

This Kubernetes base is a managed Kubernetes solution from the cloud providers. We also looked at a self-managed Kubernetes solution, but we think it's too difficult for the developers. We also looked at Pivotal Cloud Foundry, but we believe it does limit the development community too much. So this is the best fit we think.

We have Helm charts, Jenkins pipelines and, Azure DevOps pipelines to land on the cloud environment. We spent lots of focus on security and also container security. We've got Vault as a secrets management tool and also a compliance as code methodology based upon the OPA framework.

Policy enforcement with open policy agents

All kinds of departments, all kinds of people will have requirements with regard to compliance. Compliance officers, cloud platform teams, regulators, senior management. These policies can be stored within OPA. That will happen as code as well. Its policy will be a piece of software. Via docker pipelines, infrastructure as code pipelines, Helm pipelines, we can use the OPA policies to check where the right things are generated.

Infrastructure as code

How does this work for Terraform? We're going to use Terraform for infra as code. Currently, we're still using open-source Terraform. We will use Enterprise going forward. We need to see when we're going to use it. Currently the Azure teams and the AWS teams are using our templates and Cloud Formation as well. But if we want to scale—if we want to standardize—we need to have a cloud-agnostic solution for it.

How does it work? We have a Terraform pipeline built. The infra as code will be stored in Bitbucket. It's a solution we already had so we reused that solution. Via Jenkins pipeline, the artifact zip file will be generated. Via the code pipeline, Terraform will be triggered. Before a state has been generated—before a piece of software has been generated—we will check it with policy rules in OPA.

Those policy rules are also stored in Bitbucket, and there is a pipeline to copy the rules into an S3 bucket. Once the infrastructure is according to the policies, then the infrastructure will be generated. If it is not according to the policy, the build will fail, and the developer is forced to fix the issue.

If there are warnings of false positives are still in place, we might report them via Splunk. Then people can see it and judge whether it's an acceptable solution or not. Within ABN AMRO the state also should be stored in service now as a CMDB tool.

We have this pipeline in place now. I think we have four or five policies in place. Terraform is now used for creating Vault clusters and Twistlock clusters. But going forward it will also be used by the development community itself. Because if you use infra as code the responsibility to deliver the infrastructure will be part of the DevOps team. It is a shift from the old fashioned infrastructure department to the development environment.

Container security

This is what we want to prevent within ABN AMRO—a broken container, a vulnerable container. Many developers think that if they use containers it's safe, it's secure. But in real life, you need to take care of your security, both in the build environment and in the run environment.

We have procured Twistlock as a solution to manage container security. We did the RFP last year. Twistlock was implemented in the first quarter. We already did some checks on third-party images. From these checks, you see there are lots of vulnerabilities in third-party packages, so we created lots of policy that-third party packages need to be secure. We're working with our procurement department to make sure that if a third-party package is procured going forward that it is secure. And we have created a Dockerized pipeline in which container security is being checked.

In the second quarter, Twistlock was rolled out in the organization. We have scanned all containers which were running in production and teams are now checking what kind of vulnerabilities they have—what the false positives are, what really needs to be fixed.

We are creating standard guidelines and rules about the most crucial issues. We also creating portal gates now to make sure that going forward, no critical issues are implemented in production.

Next to this, we are also preparing the runtime environment rollout. So Twistlock defenders are rolled out, and we are now working with teams to check the runtime. And—very important—we are setting up a triage process. We would like to check for false positives using a similar process to the one we have for checking container security. The same process we have in place for Nexus Lifecycle, for Fortify.

And—very important—communication, knowledge, and training. This is something you need to do continuously. Every two months—every three months—we have a knowledge-sharing session for the whole development community, teaching them about Twistlock—about the progress made, about vulnerabilities. We also stimulate teams to share knowledge with each other.

Secrets management

Since we're moving to multi-cloud, we also need a solution for secrets management. Now, development teams, they were experimenting with Vault in the past. The SecOps teams they were interested in Conjur from CyberArc. CyberArc is successfully used already in our on-premise environment for Privileged Access Management.

We had some differences of opinions, so we started to work together, and we created an RFP—a request for proposal. We described the functional requirements, the non-functional requirements—all kinds of other requirements. We send them to both CyberArc and HashiCorp. We conducted the proof of concept—we described the proof points and based upon the RFP outcome and the proof of concept it was decided to procure Vault.

It was a tough decision because initially we were not aligned within ABN AMRO. The development community—they were looking in this new world—the SecOps department they were used to the old world. The lesson is you need to teach the people who are used to the traditional environment. The developer who's working every day with the containers understands the new world better than somebody who is not doing that.

Finally, we agreed to do Vault because of these reasons: Mostly the development community buy-in. It's open source, it's continuously updated and proofed. Dynamic secrets was a very important decision-maker. After it was decided to use Vault, we made a contract with the HashiConf sales manager, and we started to implement it within our ABN AMRO environment.

Results and next steps

This team has been created within ABN AMRO. It's a team consisting of both SecOps people and people of my team. So it's a good example of cooperation between SecOps and DevOps teams within ABN AMRO. We acquired all the approvals. If you would like to move an application to a public cloud, you need to have a cloud approval—you need to have a risk assessment done. All the risks that are coming out of the assessment need to be mitigated, and we have set up a team that is fully operational right now.

The current status is that a development environment has been created. A test environment also has been created—I was informed—it's based on Terraform. Within ABN AMRO, we implement it in such a way that teams can use it in a scalable way.

We could have implemented Vault a few months ago, but we really would like to have a good foundation going forward. Meaning that teams can onboard in an automated way—that all security issues are being managed. Meaning that pipelines are in place—automated onboarding is in place.

We expect that we can go live at the end of this month—may be in August. We expect that the first applications can go live after the summer holiday to make sure that we have a stable environment.

What are the lessons learned from this journey?

Governance

First of all, you need to have good governance in place. I already mentioned about it. It needs to be clear which department is doing what. We need to have shared responsibilities in place and to work closely together.

Old world vs. new world

You need to invest a lot in making the people of the traditional world aware of the new technologies because the new world is a totally different piece of cake than the old environment.

** Learning**

Learning is very important. I already mentioned about it. Teams and people need to learn on a continuous basis.

Iteration

Start small and improve little by little based on learning. Sometimes you see that people or teams think that they can make a big step at once—that they create a very big plan. It's going to fail. It's new. You will see all kinds of issues popping up, so start small and improve little by little.

Automation

And the last piece is automate everything you can. If you have manual actions. If you have manual world dependencies. If you depend on manual approvals from other teams, you will fail. You will never be able to scale. You will be able to manage it for 5 or 10 or 20 teams but if you want to roll it out for 300, 400 teams, you need to automate everything.

If you have further questions, contact me via email, via LinkedIn. Dominik de Smit is part of my team. He is also leading a DevSecOps community within the Netherlands. If you want to learn more about DevSecOps—if you want to learn more about security—you can contact him to become part of the community.