The Path to Terraform 1.0

In this talk Terraform Core Developer Kristin Laemmert pulls back the curtain on HashiCorp's process for determining what makes a 1.0 release. The talk also gives some estimations about what a Terraform 1.0 release might include and how many more versions we might go through to get there—with major caveats, of course. This is by no means a confirmed roadmap and any statements in this talk are subject to change.

That being said, our philosophy for what constitutes a 1.0 is quite clear and requires no caveats.

Transcript

Hello. Welcome to HashiCorp's 1.0 Philosophy: Setting Out on Terraform's Journey to 1.0. This talk details:

• HashiCorp's criteria for a 1.0 release

• How that differs from the traditional definition of a 1.0 release

• What I think that means for Terraform.

My name is Kristen Laemmert and I've been a developer at HashiCorp, working on Terraform open source for two years now. This is my third time speaking at a HashiConf and I'm super excited to be here.

So, a little bit about me. I live in Florida, which is in the Southeastern United States. Florida has really beautiful native fauna, such as the alligator, or swamp puppy, if you will, herons, and this super rare water loving domestic house cat. This is my cat Li Shu shortly after falling in the pool. I would like to say this is not the first time she's done that. So I don't really think it's an accident. I have some more cat pictures here on the next slide, because I don't want to disappoint folks who have seen my previous talks and are here for these photos only.

A Window Into HashiCorp Thinking

So what are we talking about today? This talk is not about features and schedules for Terraform 1.0. Sorry. Instead, I'm here to share my insights into some of the ways we at HashiCorp think: How we research, and make difficult decisions while planning releases. I think this will give you a really great look at what it's like to work at HashiCorp. And HashiCorp is just really different from anywhere I've been before.

I was inspired to share this talk after HEX 2020. HEX is "HashiCorp Exchange." Our annual all-employee in-person conference. My team, which is five engineers, our engineering manager, and our product manager, all sat down to talk about what Terraform 1.0 might look like. Now, this wasn't our first conversation about a hypothetical 1.0 release, by any means. That's a topic that's come up plenty of times in my two years here. And it wasn't a new topic then, either.

What made this conversation particularly exciting to me was our focus. Previously, we tended to talk in terms of bugs, features, and tech debt. But on that day, we took a step back and asked ourselves, what does 1.0 even mean? What does it mean to HashiCorp? What does that mean for Terraform? Can we write a definition of a Terraform 1.0 release, that doesn't specify any features, but can be used to guide our planning going forward.

A HashiCorp 1.0 vs. a Standard 1.0

And so we did exactly that. We took our starting cues from this excellent Packer 1.0 blog post. Which is widely shared as our canonical definition of a 1.0 release at HashiCorp. It was the most exciting and engaging planning session I've ever attended. And how often do you hear "exciting" and "planning session" in the same breath?

I do want to make a quick note. I'm talking about my team's planning process. It's important that you remember that I'm speaking from my own experience. So individual opinions may vary. What I'm trying to say is that I'm talking about my team, but I don't necessarily speak for the team.

We have the typical definition for a 1.0 release: "The software is feature complete and production ready."

A HashiCorp 1.0 release is very different from this Wikipedia definition shown here. If you're not familiar with our products, you might not recognize the immediate difference. But this definition doesn't apply at all. We pride ourselves on our very high standards for a 1.0 release. One of the many ways that we stand out is that our definition of 1.0, which we'll see in full on the next slide, specifies that the product must already have years of successful deployment in production.

I'm calling this out separately, because I know it can be really uncomfortable to people who aren't accustomed to HashiCorp's take on versioning. You'll find that all of our tools started their lives running in production well before their 1.0 release. And the industry has come to accept this little quirk of ours.

The HashiCorp Requirements for a 1.0 Release

Here, we have HashiCorp's requirements for a 1.0 release.

1. The project is deployed broadly and has years of production hardening.

2. Major use cases are understood and well-supported.

3. User experience of the project is well-defined.

4. Technical architecture is stable and mature.

These are taken directly from the Packer blog post I mentioned earlier. Although I have changed the order of the items, just to keep the talk rolling along smoothly. I'm going to dive into each of these points, giving a more detailed explanation, before I go into how I think about meeting that requirement in Terraform. Then later on in the talk, I'm going to take a deeper dive into where I think Terraform's path to 1.0 might take us. And I'll end it to look at our immediate future.

HashiCorp Principles: How They Relate

But first, I want to take another step back, and talk about HashiCorp's principles. Just as our release planning is guided by our definition of 1.0, our definition of 1.0 is guided by our principles. While all of our principals are reflected in the work we do, "beauty works better" is front and center in my mind, both during the release planning.

More than just natural beauty, making something beautiful is an active choice. The Terraform open source team chose thorough attention to detail, and a consideration of our peers, users, partners, and customers. We chose to put in extra time and hard work now. Not just to make Terraform 1.0 a beautiful release we can be proud of, but to even make the feature planning easier on us in the long run. So I may not call this out directly, but I hope you'll see this principle reflected again and again throughout this talk.

Production Readiness

Our first requirement for a 1.0 release at HashiCorp is production readiness. Specifically requiring that the product already has years of production hardening. I started with an easy one, because we got this. All right, long distance show of hands. How many of y'all are using Terraform in production already? Don't be shy. Raise your hands. Oh, see, that's what I thought. Look at all those hands.

Terraform was released in 2014. And it's been used in production by thousands of individuals and organizations across the globe for those years (See some of the Case Studies). I and many others trust HashiCorp, and Terraform, with my production infrastructure. We already meet the first requirement. So let's just move on.

Understood and Well-Supported Use Cases

Next. We require that the major use cases are well-understood and well-supported. Pretty obvious on the surface. We understand the use cases, because we wrote the tool to do a thing. But pragmatically speaking, you don't really know what something is until you release it. Use cases are defined by the users. Terraform's behavior can be really emergent at times.

We give users tools, and we see what we do with them. That helps us understand what the tool even is. So for example, many of the language changes we introduced in Terraform 0.12, were designed to support the clever, and entirely unexpected, ways that people found to use Terraform.

You told us how you want to use it and we adapted Terraform to meet your needs. We rely on a wide range of people. Our users, product managers, researchers, developer advocates, and all sorts of engineers, to help us codify the real world use cases. Drawing both from their own experience, and from working directly with other Terraform users.

While it's critical that we listen to what our users want and need, we also need to draw lines and say what Terraform isn't. Sometimes we have to make a difficult decision, and say that Terraform cannot support certain features well enough to meet our standards. So overall, I think we're on track here. We understand how individuals and organizations use Terraform. And the work we do directly support those major use cases.

Having said that, there are some frequently requested features that we'd like to implement. And they won't necessarily all make it into 1.0. There are also some glaring inconsistencies. Workspaces, for example. Who is using workspaces? And how? I don't know. What is a workspace? Did you know that workspaces in Terraform Cloud, and workspaces in the Open Source CLI, are two different things? That's a pain point. We're going to bring that up again later.

Well-Defined User Experience

Beyond supporting the major use cases, we also look closely at the user experience and workflows. So not only do we require that those are well-defined, but also that it is easy to learn, and it enables the major use cases in practice. Keep in mind that Terraform cores users aren't only open source users, working on a local workstation. The Terraform CLI tool runs in Terraform Cloud, and the on-prem enterprise offering. So we have a wide range of users approaching Terraform from very different viewpoints to consider. Terraform Cloud is a user of Terraform.

So as I said previously, there are definitely some workflows that aren't living up to their potential. I just mentioned the workspaces, and that confusion between the CLI and Terraform Cloud. Likewise, not all operations that you can use locally are available when running in Terraform Cloud just yet. And that's something we're addressing.

As for it being easy to learn, I'm very lucky that HashiCorp has an amazing education team that does really great work ensuring that Terraform is easy to learn. They maintain our Learn website, they run workshops and training, and publish white papers and other documentation. So we've got a lot of great people helping us make sure that Terraform is a welcoming and easy-to-learn tool. So keeping the whole user experience in mind is really incredibly helpful for us in deciding what we need to do next.

Mature Technical Architecture

Finally, we consider the overall maturity of the product. This is a very big topic. And it's kind of hard. As a Terraform user, I trust Terraform's maturity and stability. As a Terraform developer, I have a slightly different opinion. So let me share my favorite analogy, that I used during Terraform 0.12.

Terraform 0.12 introduced an entirely rewritten configuration language, HCL2. In previous talks, I compared that task to replacing the foundation of a house while there's a dinner party going on inside. It was hard. It took a long time and we had to make some really difficult trade-offs in order to release it. As it was, it took nearly a year longer than we expected. And things are still pretty complicated today.

So many of the features that we've released recently, and some of those ones that we really want to get out before 1.0, require a big amount of refactoring. On top of that, I think every Terraform developer has a pet project in mind. Some piece of tech debt, or a super important feature that they really want to address to benefit both the users and the developers. So the hard part for us is choosing which projects will have the biggest impact, and be something we can provide in a reasonable amount of time.

Then there are other components that we have to consider when evaluating Terraform's overall maturity. Are there features we plan on deprecating? If we need to remove something, we have to do that before Terraform 1.0. Unless we're willing to wait all the way to 2.0.

Terraform is more than just a tool, of course. We have internal workflows and processes that aren't all as mature as they should be before we released Terraform 1.0. We've been steadily improving our issue triage process—both issues and community pull requests. We want to improve our public communication and maybe get better at sharing our roadmaps, really just including users in our processes.

When I'm thinking about Terraform's maturity, I'm looking at not just the technical architecture. But also how we support Terraform users, as well as our internal processes and external communication. And then finally, I think about features. So we measure all of these things when deciding if Terraform is sufficiently mature for a 1.0 release.

This is one of the most complicated and exciting challenges. It's super easy to list the packages I want to refactor or pain points I want to solve. But we can't do everything we might like to, and there's a lot of work we need to do kind of behind the scenes or outside of the codebase. Each of those tasks has to be evaluated to verify that it's actively moving us closer to 1.0.

Getting to 1.0

So now that we've talked about HashiCorp's 1.0 requirements, how do we get there? As I mentioned before, we depend on a wide range of resources and people to help us research and understand the major use cases and the user experience. We take that information and we use it to refine our plans and designs, drawing firm lines to restrict the 1.0 scope to something we can actually complete in a reasonable amount of time.

Okay, I'm being super vague. But really, how do we get to 1.0? I know you want specifics. So I'm going to share some thoughts, but I need to make a really strong warning. Nothing I'm about to say is a promise. We have only just started talking about Terraform 1.0. And even then, the conversations have focused on what tasks we want to complete before 1.0. In a 0.14 release.

Terraform 1.0 Themes

Remember that I'm really here to give you examples of how we talk and think about release planning. I don't have firm plans to share. But having said that, I can show you some of the themes that I see coming up on our way to 1.0.

Architecture

We can improve Terraform's architectural maturity by addressing tech debt. The command package is kind of a personal frustration of mine. There's a lot of boilerplate that needs to be copied into each command. So if you fix a bug in one command, you'll probably need to remember to go fix it in six more. That drives me crazy, and I really hope to get some time to improve it. Although I need to be honest. I kind of doubt that this is going to be prioritized for 1.0.

More valuable, I think, is improving the test writing experience for developers. Terraform's internal data structures, such as the plan files and state files, are really hard to handcraft for a test. And most packages inside Terraform have slightly different helper functions, so they all do the same thing, but slightly differently. So making this easier for developers, for me, would be a big win with a clear 1.0 readiness benefit. It's much easier to trust the maturity of the code base when you can trust the maturity of the test suite.

Another big project we've been discussing is kind of a gnarly refactor of Terraform's internal plan-refresh-apply walk. This may be invisible to users, but it's going to make our life easier and really help us implement some improvements, and maybe fix some bugs that people have been asking for.

Support

We're reviewing our supported platforms, deciding if there's any that needed to be added or removed. We're also ensuring that our support policies (LTS) are in place, and communicated to users. And of course, once we have a 1.0 release, we're not going to allow any breaking changes into 2.0. So if there's anything we really want to break, the time to do it is now.

Internal Processes

We're also reviewing our internal processes. So right now I have a giant pile of GitHub issues, somewhere around 1300 issues. What I need is a groomed backlog that I can reason about. I can use it to identify themes and pain points. So this is something we're working on. And specifically, I want to thank my manager, who's taken on a huge chunk of the issue triage. I'm really grateful. So thank you, Daniel.

It's still going to take us some time to get that nailed down. We're also working on actual release planning, like the kind that comes with estimates and dates. So there is something my manager has been begging us for. That's hard work. It takes a lot of practice. Probably no surprise that it's not my favorite thing to do, but we will get there.

Communication

We have some really big ideas for improving our communication with the community. Many of which are being driven by our product manager, Petros. Hi, Petros. We would love to share roadmaps and milestones publicly and have a public process to talk about feature requests so contributors know exactly what kind of pull requests we're interested in seeing.

Possible Features

Finally, I think about what features we'd like to add.

• Feature parity between the CLI and Terraform Cloud is one theme that I mentioned already

• Improving Terraform's handling of sensitive values

• Improving Terraform import

• Improving module testing.

All of these are features that come up frequently. So a number of people at HashiCorp are helping us research these features to determine what we should focus on. Interviewing customers, soliciting feedback from the open source community. We absolutely depend on all of those people, and all of you, to help us reach 1.0.

We also look at the most upvoted, GitHub features and issues. So when I say upvoted, I mean, we count those little thumb up responses on issues, and we compare those issues to our 1.0 criteria. For each issue we ask:

• Is this something we want to fix or implement now?

• Will it positively impact a large proportion of our users?

• Is this something that we have to get in before 1.0? So is this like a breaking change? Or is it something we can do in a minor release, if necessary?"

Where We Are Today: Terraform 0.13

Now, you know how I think about planning Terraform 1.0. Starting with HashiCorp's principles, and using our requirements set out in that Packer blog post as a guide. I hope you've found my insights interesting, and have learned a bit. Not just about how much work goes into planning, a 1.0 release at HashiCorp, but how little of that work involves planning features. How much of it focuses on these bigger, almost philosophical questions about Terraform's overall maturity, and what we want to build.

Terraform 0.13 is available right now, in beta. With a tentative release planned in July. Terraform 0.13 introduces module expansion. So you can use for_each and count with modules. You can now also use the depends_on meta argument to handle hidden dependencies and modules that Terraform can't automatically infer. There's also a new provider source attribute, and this has been my project for the last year, which allows you to specify alternate registry hosts for providers. And it sets the stage for community providers in the registry.

We've also added some options for the provider installer, which lets users configure file system paths where providers can be installed from. And we've got some more options for that planned for future releases.

Terraform 0.14

We will most likely have a 0.14 release. That's our last chance to deprecate features or introduce breaking changes, before 1.0. Will we have 0.15? That's anyone's guess at this point. I honestly don't know. What I can say is, anything we release is going to be driven by the requirements I just shared, chosen specifically to get us closer to releasing Terraform 1.0.

Terraform wouldn't exist as we know it without our wonderful community. And so I'd like to close this presentation by encouraging everyone here to get involved, in whatever capacity works for you. Test out the beta, ask questions in our forums, open an issue in GitHub, and let us know what you'd like to see in Terraform 1.0. Thank you all so much for listening to me today. [Kristin picks up her cat] I'm so sorry I couldn't do this in person. And I'm incredibly grateful to HashiCorp for giving us all this opportunity to participate. Thank you.