Workload orchestration never sleeps

Seeking stability in a volatile environment

Digital banking has reshaped the world for private consumers and big businesses alike. Today, users can make deposits, issue payments, open accounts, and apply for loans from virtually any device at any time — and the Q2 Platform powers those solutions.

Q2 provides the backend infrastructure and systems to more than 800 financial institutions and fintech companies that enable exceptional financial experiences for their customers. Today, the company is the driving force behind 10% of digital banking experiences and processes financial transactions that equate to roughly 5% of the national gross domestic product (GDP).

But as the company's customer roster grew and the range of use cases evolved, so did the need for replacing its time- and resource-intensive manual infrastructure provisioning practices. Keeping those systems online and up to date is a challenge.

"In the digital banking and financial services world, high availability and rapid delivery of new solutions is imperative for remaining competitive and relevant," says Jordan Hager, vice president of hosting architecture (including DevSecOps, site reliability, & architecture). "We needed a way to simplify our software deployments and build a continuous delivery model that would help us stay ahead of evolving market and financial services trends. In our industry, we look at technology as the great equalizer that allows community financial institutions to compete."

Time is money, and manual approaches take both

Since its founding in 2004, Q2 has served the financial technology needs of more than 18 million consumers. Firms ranging from well-known international banks to innovative fintech startups have relied on Q2 to help them deliver the features, functionality, and engaging experiences their customers demanded in exchange for adopting the new services and creating additional revenue streams.

For years, Hager's team, including Site Reliability Engineers Bijan Rahnamai, Daniel Dreier, and Team Lead Cody Jarrett, manually provisioned and deployed infrastructure required to deliver Q2's products. Any new feature or service request from customers or the internal support team required the IT team to selectively assign server space based on available resources, priority, and other factors. But the intricacy and increased velocity of the requests made keeping up with demand increasingly challenging.

More importantly, the existing processes didn't allow developers to prototype and deploy new applications or update existing ones without risking costly downtime. "On a peak day, our platform processes as many as 1.2 million logins per hour. COVID-19 has accelerated the adoption of digital platforms, and our solutions are as critical to consumers and small businesses now more than ever. Downtime is very costly, and our customers expect anytime, anywhere access to our systems." Hager says. "Manually orchestrating the moving pieces not only makes quickly bringing new features to market more difficult, but it also creates a host of problems with monitoring the health of different services to make sure they're actually working."

Q2's first-generation orchestration platform was an in-house developed product that the team quickly realized wouldn't meet its growing need to scale. "Building a new environment with the tool still took up to a week just to balance network loads and provision applications and didn't address any of the health monitoring and performance management features we wanted," he says. "It was pretty clear we needed to reset our expectations and find a system that could simplify our deployment operations and automate health checks on all our applications."

Q2 is using HashiCorp Nomad's workflow orchestration for higher availability and to deploy applications and upgrades in an efficient, scalable way.