Lessons Learned From the Hypergrowth of HashiCorp's Remote Engineering Teams

In this keynote from the final day of HashiConf 2019, Preeti Somal, the VP of Engineering at HashiCorp, shares some of the lessons she learned in HashiCorp's current period of extreme growth regarding remote-first engineering culture and following the company's longstanding set of principles.

Early on, HashiCorp had engineering groups for each product that functioned will with just an Engineering Lead and an Engineering Manager. As teams product teams grew, engineering teams were further subdivided into workstream groups (e.g. Terraform Cloud, Terraform CLI, Terraform providers, Terraform Enterprise)

Each workstream unit includes: - Engineering Lead - Engineering Manager - Designer - Product Manager

Later HashiCorp added Engineering Services, which supported all workstream groups and helped relieve their pain points. We did this by embedding Engineering Services team members in each of the product teams.

Watch the rest of the video to learn more about how HashiCorp has also focused on design, adding a new UX research team, and security by hiring a CSO. You'll also learn more about our remote-first culture and remote hiring.

Transcript

Good morning everyone. I hope you're enjoying the conference as much as I am and had a wonderful time last night as well. My name is Preeti, and I have been a part of the amazing HashiCorp engineering team since January 2018.

Prior to that, I was at Yahoo running Yahoo's cloud services. We were building mission-critical, highly available, highly scalable services to run Yahoo's business. We were using HashiCorp products, although I wish Consul 1.6 had shipped five years ago. It would have been life-changing for my team back then.

When this opportunity to lead engineering at HashiCorp came up, I was intrigued. The role of the HashiCorp products and the criticality in using them in our journey to the cloud operating model was clear to me, given my vantage point at Yahoo. But it was undeniably the people I met and the culture that I experienced during the interview process that convinced me of my fit for this role.

I've never looked back. It's no surprise that for my debut HashiConf talk, I'm going to talk about our people and our principles. I'm going to take you on a journey into the growth of our engineering team. I’m going to talk about how the challenges we faced with our rapid growth have shaped our organization and its evolution. I'm also going to talk about our remote culture.

A distributed engineering team

Most of you probably know engineering at HashiCorp is remote first and fully distributed. We've learned a lot of lessons along the way. This morning I hope to share some of those lessons with all of you.

First step. Why are we here? Our mission at HashiCorp is to enable customers on the journey to the cloud operating model. You heard a lot about this yesterday. I'm going to skip right over this and get into my story.

First, I wanted to have a quick note on the visuals that I'm using. As I was starting to work on my slide deck for this keynote, I wanted to find a visual metaphor. One that would help me illustrate our story and would also be true to the ethos of our culture. Several options were considered and discarded until I landed on a Lego team.

You see, we at HashiCorp, are builders—and what's better than Lego to represent and illustrate our story? I hope you enjoy seeing the visuals as much as I've enjoyed using them in putting together my slide deck this morning.

Our first picture, of course, is Mitchell and Armon. Our story today starts with Mitchell and Armon right here in Seattle. I'm not going to comment on who's who—we'll leave that at that.

Mitchell and Armon were busy toiling away, building tools to fill the gaps in their toolbox. They realized that the tools that they were building had broader applicability and started to open source them. This open source resulted in a community that drew like-minded engineers that thrive on solving really hard problems. Slowly that community grew into a village.

This was a village around a number of tools that the open source community was producing. Over a short time, the tooling grew to solve a broader set of challenges. This resulted in the creation of six projects and followed pretty quickly thereafter in the commercial additions where we solve a lot of problems and challenges at scale and enterprise complexity. You heard more about this yesterday.

The rapid adoption of our open source and enterprise tooling was resulting in significant growth in our engineering team. Here's our engineering team, it's growing large—and we've got these six products that we're building out. At this point, we started working towards how do we organize ourselves? How do we try and bring more structure into our engineering organization?

That village was now growing into a city. This is a city that had distinct neighborhoods. Each neighborhood was focusing on a product. Along with this growth—and the creation of teams that are focusing on a product—roles and responsibilities were also emerging.

The engineering manager and engineering lead

Two key roles emerged at this stage of the company. The role of an engineering manager. This individual was responsible for the wellbeing of residents in their neighborhood; making sure all obstacles towards the delivery of products were being removed—as well as coordinating with other neighborhoods to ensure that the overall city was functioning as effectively as possible.

Along with the engineering manager, the role of an engineering lead emerged. Each of our products has an engineering lead. They work with the residents of that neighborhood to make sure that everything is lining up well and the overall design holds true to what we are trying to achieve with our products.

This worked well for a while. At this stage, we had teams that were focusing on a product. We had the engineering managers and engineering leads. Then in the mid-2018 timeframe, our rapid growth spurred the next stage of our evolution.

Workstreams

You can think about this being similar to what happens when a large number of people influx into a city and that city has to evolve to avoid gridlock and to continue to function effectively. That's what happened to us with our rapid growth. There were a number of elements of our next stage of evolution, and I'm going to walk you through those next.

The engineers that were working on a single product became too large to work towards an effective single team delivery model. This was spurred by our Terraform team, which is by far the largest engineering team within HashiCorp. We had to think about how we structure this so we can effectively continue the pace of delivery while growing the teams.

We hit upon the concept of a federated delivery structure. We call it a workstream. We have several workstreams for Terraform—for instance, one for the CLI, one for Terraform providers, Terraform Cloud and Enterprise.

Each workstream is a self-contained unit with an engineering manager, the eng team, an eng lead, a designer and a product manager. They are empowered and enabled to work as independently as possible so we can continue that rapid delivery.

This only works when you've got logical parts of the product area that are owned by a workstream. Workstreams were the first evolution spurred on by this next stage of growth. This is an example of a picture showing the Terraform workstreams.

At this time, we started to feel the pain around the lack of infrastructure. This is what happens when roads aren't maintained. Trains don't run on time. Maybe traffic signals that don't work, and accidents start to happen. These accidents result in costly time being spent on recovery and result in more of a reactive way of operating. This is not where we wanted to be.

Engineering services

At this point, we started working towards an engineering infrastructure function that we call engineering services. The first question was, how do we structure this?

Those of us who've been involved in any engineering services or platform type projects, know that things can go horribly wrong. We knew—going in—that we did not want to build a bridge to nowhere. We needed to make sure our engineering services function was going to be effective in helping relieve the pain points that the engineering organization were starting to see.

We explored several models. We landed on a model of embedding engineering services team members into each of the product teams. And there—that person is the engineering services member.

This led to the engineering services team understanding and living the problems that each of the product teams had—proposing solutions that were pragmatic and would help build us towards a broader picture. After all, pragmatism is one of the key elements of our Tao as well.

The tooling we use in our engineering services function is consistent across all the product teams. But the workflows are tuned to how each product team operates. Workflows, not technologies, are a part of our Tao. Our engineering services function is an example of how we effectively use that, even in our internal offerings.

I'm happy to report that this organization is growing well. We've made a lot of progress in alleviating some of the pain points that we were starting to feel about a year ago.

Design

The next element is design. You may be wondering what this picture has to do with design. Let me zoom in a bit and show you that building over on the left. It looks out of place with the rest of the city.

One challenge in a multi-product company is how do we make sure that our products don't reflect our org charts? Usability is really important to us. You heard Armon talk about this yesterday. ‘Beauty Works Best’ is one of our principles, and we have a huge focus on making sure that our products are usable.

This focus manifests itself in several ways; from the single binary model to the focus on simplicity for language and API, to the increased investment in Graphical User Interfaces, our documentation, and so on. Hopefully, all of you agree that you can see this focus on usability from HashiCorp in your day-to-day lives as you're using our tooling.

Everybody in engineering is responsible for usability but we also lean on a team of experts—our design team. Our design team helps and guides us as we are working towards making our products more usable.

One aspect of our design team—a function that we recently added—is the research part of design; what we call UX research. They're here on the floor. Hopefully, some of you've had a chance to meet them. I was told that yesterday after the Terraform Cost Estimation announcement, roughly 50 members of the conference attendees were able to go over and do a user test of that new feature.

If you've got time today, find them out and do talk to them. We are excited about the user research function being an addition to our design program. This will let us get more data-driven insights into gaps and areas that we need to focus on.

Security

The role that our products play in delivering mission-critical applications places a huge responsibility on us—to make sure that the security of our products is world-class. We take this responsibility very seriously. We've been building out a security organization with a chief security officer that came on board early this year. Hopefully, you've had a chance to meet some members of our security team.

This team does what it takes to make sure that we are upholding the trust that our customers have placed in us. Some examples include: Working with product teams around security design, enabling the ‘secure by default’ mindset; handling inbound notifications as well as responsible disclosures—working with external consultants around various levels of security work that we do, and more. I just want to make sure everybody recognizes that, for us, the responsibility around making sure our products are world-class, secure, is something we take super seriously.

I've covered these elements; workstreams, engineering services, design, and security. I am going to move to remote next, and it may be time to leave our Lego friends behind. But first, the irony of using a physical metaphor—that of a village, a city, and neighborhoods—to describe an organization and its evolution when that organization is 100% remote and fully distributed—was not lost on me as I was building these slides out. I often paused and chuckled and said: “wait, did I do that?” And yes, I did. I think it worked fairly well. Hopefully, some of you will agree. But it's time to talk a little bit about remote.

Remote working

That is not who we are: We do not go into an office. We are not frustrated with our coworkers. This is who we are: We work in Google Docs, in Slack, GitHub, in Zoom. This is another visual of who we are.

Our roots are in open source. The foundation of our engineering organization is built upon our open source roots. Most of our work ends up happening in GitHub and shared documents—and Slack and Zoom are our office space enablers. It's not uncommon to see pictures like that. That's our marketing team, not our eng team—but that was the best visual I could find. Our All Hands look like this on Zoom, and we get a lot of face time as well as interaction using the tools that we do.

Hiring

I've talked a bit about tools. I'm going to talk a little bit about hiring.

Today we are able to hire in 35 States in the United States and five countries internationally. We are mainly gated on hiring across the world based on local employment laws. One myth that people often have about remote companies and remote work is that you can hire people anywhere in the world, and that's not entirely true.

If you're interested in learning more about that, Mitchell has an awesome Reddit thread that goes into more detail—definitely check that out.

As part of our interview process, we put a lot of focus on culture fit. That’s a screenshot out of our interview tool where the interview panel—as they're talking to candidates—is assessing how characteristics and attributes match our principles. We also have a lot of upfront conversations with candidates that may not have remote work experience to make sure that they're reflecting on whether or not remote is right for them.

We recognize that remote may not work for everybody and we want to make sure that when people come to HashiCorp, they've thought through it because we want them to build their careers here. We want them to stay here—we want them to do the best work of their lives here, and we want them to internalize what remote work might mean for them.

It's pretty common for our recruiters and our hiring managers to have that conversation about what remote means—and think hard about it. It's not all ponies and unicorns. That's a little bit about our hiring process.

Onboarding

Once we've hired, the next stage is onboarding. We think about onboarding in two phases:

The first phase of onboarding is driven by our people team. In this phase, all the new hires over a certain time come to San Francisco, and they spend a couple of days meeting with all the company leadership—engaging in conversations, learning in more depth about the company, our business, our history and what each functional org does.

They're also building bonds with their onboarding class. I still remember the members of my onboarding class and I meet up with them pretty regularly. This is a screenshot from one of our more recent onboarding sessions.

The second part of onboarding is where the engineering manager will work towards onboarding the engineer into the workstream that they're going to be a part of. That's a very well thought through and well-crafted plan. It's customized to the engineer who's onboarding because we see a huge variance.

We've seen people coming on board—they've been users of our products, they've been regular contributors to our products. For them, it's a pretty rapid ramp. It's not uncommon to have an engineer do a demo at an All Hands within the first month of them starting with us. That's how fast the ramp can be.

Staying connected

That's onboarding. Now we've hired you, we've onboarded within engineering. The next question is, how do we stay connected? We don't see each other physically every day, but we've got the tooling in place. That's not necessarily enough. We've built a number of practices and traditions where we keep connected.

Our need for collaboration is clearly the strongest at the workstream level. That doesn't mean that we ignore the need for communication across all of engineering—or the entire company. We put a lot of thought into that.

Within the workstream, for instance, we do asynchronous Standup Updates, and they're always fun to read through. I read through them pretty much every day. Across the company, we have something called a Chat Roulette where people get randomly paired and, they spend half an hour getting to know each other.

Within engineering, we have a Weekly Lightning Talk. The topics for this talk are solicited via a CFP. This is an amazing opportunity for our engineers to talk about things that they're passionate about and start getting some experience about public talking in a very safe environment. A lot of engineers are working towards a goal of doing conference-level talks, and this ends up becoming a nice way to start thinking through that.

We also do team offsites. I know pretty much every product team has done an offsite where they'll spend a few days co-working and building that ‘being together in a physical space’ bond.

Our company has an All Hands every week. We rotate the agenda for every functional organization. When it's engineering’s turn, we mostly have engineers go on stage and give demos of features that they're working on building. It's a great opportunity for them to show the work they're doing—and for the rest of the company to get an insight into what's cooking within engineering.

You all may be wondering is there any time where the entire company gets in the same physical space? There is. It's something that we call HashiCorp Exchange. Our fabulous events team organizes it for us and—of course—it’s world class.

For the most part, we do okay. It's always interesting to bring all of us together when we don't go into an office regularly. But when we do get to HEX—or HashiCorp Exchange—we have a great time meeting our colleagues and building those relationships.

This is an image from HEX this year. You can see, it's a world-class production for an event, and we do all kinds of things. We use this time as social time, reinforcing our values and our mission, co-working, doing customer panels. The goal here is to fit things in that you can't typically do over Slack and Zoom. It's an event that all of us look forward to. Ours this year was in May—it was a fun time.

HashiCorp’s principles

It's almost time to wrap up. Hopefully, this section has given you an insight into how remote is working for us—and some of the tips and tricks that we use. For my wrap up I’ve put together some visuals of how much we've grown over the last few years. Mishra from our Dev Advocates team has been phenomenal helping me with the slide deck. Shout out to Misha. Thank you so much.

He hit upon an idea of using pictures for the traditional employee photograph after every HashiConf since the beginning. That's 2015, 2016, 2017 and 2018 and I will update this after the one from tonight.

The point here though is that we've grown tremendously as we've scaled. The one thing that has remained consistent throughout this growth is our principles. Our principles are what allow us to scale and provide an anchor in this exciting and sometimes turbulent journey.

For those of you in the audience looking at building engineering organizations—if there's one thing I can leave you with, it's to be thoughtful about the culture and the principles that you want to enforce; to write them down and to use them as an anchor when things are hard or moving fast or you're scaling as rapidly as we are.

That brings me to the end of my talk. I hope you've enjoyed it. I had to put up a Lego caricature for myself because that would be unfair to Mitchell and Armon. That's me and that's where to find me. Our user experience research team is conducting testing today as well—if you get a chance to go over. And those are some pointers to more information.

I hope you've enjoyed this talk. While we don't claim to know everything, we've learned a lot along the way, and we're happy to share our learnings with our community—and we hope to continue doing so. Thank you. Have a wonderful day today. It's been a fun conference so far. Thank you.