Nomad Community Story: How Nomad powers a $1B hedge fund in Brazil

Navi Capital, a Brazilian hedge fund, shares their story about how they used HashiCorp Nomad to streamline data-driven investment decisions.

• 2 Weeks to launch new microservice-based projects running on Nomad
• 1 Operator to build and manage data pipeline with Nomad
• 70% cloud resource cost reduction by using Spot Instances and Nomad

»About Navi Capital

Navi Capital is a hedge fund specialized in Brazilian stocks. Founded in 2018, with a 10-year success record as a team, the firm has built a reputation for its unique investment process and culture, The firm leverages its combined investment knowledge and carefully studies and analyzes rich market data to select the best medium- and long-term investments for its clients.

»Investing in the future

As an emerging global market, Brazil is ripe with investment opportunities. And hedge fund Navi Capital is helping to uncover high-return investment opportunities by combining its institutional investing knowledge with data-driven insights to evaluate Brazilian equities. Though the company has had tremendous success in selecting medium- and long-term investments, it also recognized that pulling in data from government and other public sources would dramatically improve its ability to remain competitive in the future.

In response, the firm created a data science department to help incorporate more public data into its decision-making process to mitigate the risks of poor investments. But aggregating and analyzing huge quantities of structured and unstructured data requires extensive and often expensive infrastructure that can spell trouble for a young and growing organization.

“In the hedge fund industry, research by financial analysts is critical for making good investment decisions and creating a data pipeline seemed to be the best way to leverage public data about Brazilian equities in our decision making. But with limited human and financial resources, we wanted an efficient and low-friction method for building it and realized that a well-orchestrated containerized environment was our best approach.”

Carlos Domingues, IT infrastructure lead at Navi Capital

»Building a data pipeline from scratch

As a lean hedge fund, Navi Capital can take any competitive advantage it can get. And leveraging massive amounts of public data for investment insights and decision-making provides just that. However, the company didn’t previously have a data science team and had to create both the department and the data pipeline project from scratch — including all the infrastructure to support the operation.

"Manually pulling public data into spreadsheets and transferring it to analytics tools isn’t an efficient way for our investment team to operate. We needed a more modern, cost-effective, and faster way to compile rich data and operationalize it, which started with building the right infrastructure and moving to a containerized environment."

Antonio Lobato, partner and head of the Data Science team at Navi Capital

Beyond improving availability and aggregation, Navi Capital also wanted to keep the costs of its data projects low to maximize margins and opportunities for reinvesting in its operations. Knowing that microservices orchestration and handling massive volumes of data could become expensive if left unchecked, the firm set out to find an orchestration platform that could efficiently manage its computing resources to minimize waste and maximize the ROI of its data-based investment decisions.

“In the hedge fund industry, the availability of accurate information is crucial for making sound decisions,” Domingues says. “Implementing a low-cost data pipeline was an obvious solution for making public equities data available to our investing team. And we realized the right infrastructure could make it scalable and affordable.”

»Fast decisions, smart investments, and massive cost savings

As part of building out its data pipeline, Navi Capital’s first technology decision was choosing an orchestration platform. After looking at Amazon EKS, ECS, and Fargate, the company chose HashiCorp Nomad because of its simplicity, ease of use, and reputation for high-availability and functionality.

Unlike more resource-intensive and complex platforms like Kubernetes, Navi Capital found that Nomad could automate the most business-critical workflows and unify infrastructure to maximize resource utilization, efficiency, cost management with very low operational needs. With Nomad, Navi Capital manages a robust data pipeline that’s made up of a series of containerized microservices deployed in a cluster as the control plane. These microservices expose APIs that can be used by analysts to pull reports or trigger data processing batch jobs.

Best of all, adding Terraform — HashiCorp’s infrastructure-as-code tool — to Nomad enables Navi Capital to easily manage and deploy new microservices as they’re developed. The end-to-end process has been fully automated with CI/CD pipelines. Developers can simply define the microservice deployment through Nomad job templates and the deployment will be committed via Nomad API calls within seconds. That means the company can quickly build a new microservice-based project to pull data from a Brazilian regulator or other source of rich public information that’s available for the investment team in a matter of two weeks.

“We’ve really streamlined our data operations with Nomad and freed up our time to work on more high-impact tasks,” said Carlos Domingues. “Once we launch new microservices, they just work. That means we can focus our time on developing new services that bring even more data to the investment teams, which is crucial with such a small development team.”

Navi Capital now has built more than twenty projects running on microservices that make hundreds of gigabytes of data available for financial analysts and partners to use for investment research. And this new containerized approach has actually reduced the cost of its data infrastructure, even though it was quite low to begin with. Because these internal batch jobs don't have strong SLA requirements, the company can save up to 70% of its largest operating cost -- long-running EC2 instances -- by using Spot Instances and Nomad together without worrying about availability.

“Nomad has vastly improved our investment process and reduced our costs for leveraging data,” Domingues said. “We’re excited to continue scaling our data infrastructure and launching new microservices to bring more insights to our investment team, and deliver even greater results to our investors.”

»Technology Stack

• Infrastructure: AWS (100%)
• Platform: VMs, Containers
• Proxies: Fabio
• Load balancer: AWS Application Load Balancer, CA AWS Application Load Balancer
• Provisioning: Terraform
• Security management: AWS Config + AWS IAM + AWS Security Hub
• DNS: Route 53
• Service Discovery: Consul
• Databases: AWS Aurora Serverless
• Raw Data / Semi-Structured Data Storage: AWS S3
• Data Processing: AWS Athena, Databricks Spark
• Machine Learning Model Lifecycle Management: MLFlow
• Data Catalog: AWS Glue
• CI/CD Pipelines: Gitlab
• Batch Job Scheduler: Apache Airflow
• Logging and Monitoring: AWS CloudWatch
• Private Package Manager: Nexus Repository

»Navi Capital Partner

Carlos Felipe Domingues e Oliveira is Navi Capital’s IT Infrastructure Lead, responsible for building the systems and processes that make the firm’s work possible. Prior to joining Navi Capital, Carlos spent several years working as an IT engineer, part of Huawei’s nationwide Mobile Network Customer Experience project for over 60 million users in Brazil. Carlos holds a Bachelor’s degree in engineering from Universidade Federal do Rio de Janeiro and additional international studies at the University of Bristol in the United Kingdom.

Antonio Lobato is a partner and head of the Data Science team at Navi Capital. Prior to Navi Capital, Lobato served as a quantitative and business intelligence researcher, IT project manager, and as a corporate credit control analyst at well-known financial institutions. He holds a Master’s degree in electrical engineering from Universidade Federal do Rio de Janeiro.

   Are you interested in telling others your Nomad story or perhaps how Nomad helped with that amazing thing you built? Let us know. Fill out this short survey here.