Keynote: Cost Estimation and Clustering in Terraform

The Terraform Cloud launch has been over a year in the making. It's another big step in Terraform's journey toward a more platform-like experience.

In this keynote, Terraform Engineering Director Paul Hinze and Terraform Product Director Robbie Th'ng talk about how Terraform's goal has been to borrow good ideas from the application delivery pipeline and use them in Terraform's infrastructure delivery pipeline as the two pipelines closely interact.

They also announce two new features, one for Terraform Enterprise in private beta, and one for Terraform Enterprise and Terraform Cloud Teams & Governance tier.

First, they show how the new cost estimation feature works for Terraform Enterprise and the Terraform Cloud Teams & Governance tier. There are many creative ways to use it to configure policies to preemptively manage cloud usage before you provision resources. They hope that users find even more innovative ways to use it and request new feature ideas.

Second, they introduce the beta version of Terraform Enterprise's next-generation architecture with Clustering, which provides greater scalability, availability, and easier installation and management. This has been engineered and collborated on with customers for years, and now the new foundation for Terraform has finally arrived.

Transcript

Paul Hinze: Look at all these people who showed up for TerraformConf. Welcome to TerraformConf. We're glad you had a great time, Day 1. My name is Paul Hinze. I am the director of Terraform engineering.

Robbie Th'ng: And I'm Robbie Th'ng. I'm the director of Terraform product, and we're very happy to see out the end of Day 1 of not-TerraformConf with you. It's been a good day. It's been a good first day, and we've learned a lot of things. But "Why are we here?" is a question that we constantly ask ourselves.

Paul Hinze: Every day. And also today.

Robbie Th'ng: Also today, because HashiCorp founder and CTO Armon Dadgar announced a lot of stuff this morning, a lot of really exciting Terraform stuff. We're trying to figure out what else we've got to add.

I'm here representing a team of product managers that a lot of people have probably spoken to here if you're using Terraform Cloud or Enterprise, or if you're using Terraform at all. We see a lot of broader trends, a lot of broader patterns among Terraform users. And we wanted to come and share them.

Paul Hinze: And I'm representing a ton of Terraform engineers at HashiConf working to make solutions to those problems across the CLI, the providers, across Terraform Cloud and Enterprise. And those teams are capable of producing a lot of work. A lot of software's getting shipped at this point, so much so that we have more to talk about today, even after we monopolized the morning with Terraform Cloud.

Robbie Th'ng: We're just going to keep going after the mics go off, we'll just keep talking. It's going to be really awkward. We cannot go much further, though, without talking about the big thing, which was announced this morning, which is ...

Paul Hinze: Happy Terraform Cloud launch day, everybody. This is a huge day for the team. We're so excited to roll this out. This has literally been years in the making. These workflows are workflows that we've honed in Terraform Enterprise. Last year we decided, "Hey these collaboration workflows are core to the Terraform experience. We want to get them out there for everybody."

We've been working all year to do that, and today it's here. Today it's out, for free. We're so excited to get everybody on board and using it. So, terraform.io. If you haven't checked it out yet, please leave this room, stop listening to us, and go check it out.

Robbie Th'ng: We're going to say terraform.io multiple times.

Paul Hinze: Yeah. We want to see how many times we can sneak it in: Just sign up at terraform.io.

Robbie Th'ng: If you want to learn more, there is a much more detailed session tomorrow with 2 of the excellent engineers from the Terraform Cloud team that are here, software engineer Mike Nomitch and Terraform Cloud technical lead Robert Tillery. They're going to talk about some of the features and some of the things under the hood.

If you want to know more about Terraform Cloud, definitely hear that session.

Paul Hinze: It'll be a great talk. These are 2 of the engineers that had been working on Terraform Cloud all year, and so they're the best to teach you about it.

Terraform as a platform

Robbie Th'ng: Let's get into it. We're going to take a lot of things Armon said this morning, but one thing in particular to start off with, which is Terraform as a platform. This is worth digging into for us, because it's been informative for how we think about the product, but also how our customers tell us they think about it as well.

When you're using Terraform just as a tool, if you're using it on your own and you're just provisioning a lot of infrastructure, then what you tend to find is that it is very useful. That is a good way to develop. It's a good way to deploy.

But when you start to use it as a group, when you start to use it with more than 1 person, when you start to use it as teams or when you take it to an organization, it becomes this platform. Because it becomes less about the tool itself, less about the CLI.

Or, if you're using clouds—hopefully you are (terraform.io)—it becomes less about the CLI, and it becomes much more about the processes and the workflows that you build as a part of a company or as a part of a larger organization with lots of teams collaborating. And this is really important for us to recognize, because it's quite similar to something else in the DevOps world that we've seen.

Paul Hinze: The application delivery pipeline is a general concept that's pretty well cemented in modern DevOps organizations. This is the process by which you get your code from the developers' machines out into running applications.

This is the same kind of transition that we see with Terraform, from tool to platform. You think about the base primitives of a deployment, of a continuous integration tool. They've now been lifted up in our minds to this general concept of, What is my application delivery pipeline? And what's cool about being infrastructure as code is we get to borrow all of these concepts and developments from the application development space.

Terraform is going through a similar transition of saying, "You start with that CLI tool, but then as you implement it in an organization, it becomes more than that."

It stands to reason that one of the most common questions that we get about Terraform is: How does Terraform fit into this application delivery pipeline? And we've done a lot of thinking about this and a lot of talking with customers, and we're ready to share some conclusions with you about how we think about this.

Robbie Thing: We think about it as something separate. We think about it as an infrastructure provisioning pipeline on its own. We think about it as something completely distinct to the application delivery pipeline, and I think that's important. And if you've used Terraform in a large team or if you've used it in a company, you probably know this as well.

There are things that you want to do with infrastructure provisioning that are just related to infrastructure. There are levels of control, there is a degree of separation that you want to apply, and that's critical.

If you just leave infrastructure provisioning as a footnote, as a runner, or just as a task inside a CI/CD tool, you're not going to get that level of control, and that is very critical. If you've experienced that, you probably know exactly what we mean. But you also can't have the infrastructure provisioning pipeline too far away. It can't be too separate from the application delivery pipeline.

Paul Hinze What we've seen as the best way to think about this that, by modeling the infrastructure provisioning pipeline as its own distinct entity, it allows you to focus on the needs that are specific to infrastructure.

We'll go into a little more detail about what that means. But it's something that lives very close to your application delivery pipeline, because for various use cases, you're going to want these things to interact. And you get the flexibility in those interactions by modeling them as these distinct pipelines.

Paul Hinze: Let's take 2 examples. Let's take a development environment for which you have a large QA team that you need to test changes, and you want to be able to deploy an ephemeral development instance of that application for every branch. That's a pretty intimate relationship between these 2 pipelines.

If you take it all the way to the other end of the spectrum: an incredibly locked-down production environment for which you may want the application delivery pipeline to be very far away from the infrastructure, maybe provably so for compliance reasons.

By having these 2 pipelines modeled distinctly, it allows you to express these varying use cases in a coherent way, and allows you to focus on the needs of infrastructure specifically.

Robbie Th'ng: This is important because this is where we've been. And if you're using Cloud or Enterprise or you're building around Terraform, this is where you are as well. This is where we've been trying to add value. This is where we've been working together in order to try and understand how to build this pipeline.

That is something that we want to share back. The things that we've built in Terraform Cloud and Enterprise today are a result of the conversations that we've had with most of the customers learning these facts. Like the idea of having your infrastructure broken down into workspaces, decomposing that, and having that full organization view.

The visibility is very important. Having governance and control with something like Sentinel, to the granular level that you need for your team in your organization. Again, something we learned together, something we built together.

And finally, sharing best practices, using reusable configuration that you want to have in your organization, pass it around with modules, and having a private module registry to do that. Really important part to have. This is where we want to add more value together, which leads us back to what we want to talk about today.

Paul Hinze: This is where cost estimation comes in. With this idea of this infrastructure provisioning pipeline in mind, we have 2 things we want to dig into with you today.

The first is grabbing the baton from Armon this morning and digging into the cost estimation functionality. And then we want to visit Terraform Enterprise, the place where we have really honed these ideas and discovered this infrastructure provisioning pipeline concept. And we want to update you on some of the latest developments there.

Cost estimation

Robbie Th'ng: Armon talked a bit about this morning, and we're going to cover a little bit more today. We've spoken to a lot of customers, we've spoken to a huge amount of people about cost and about things related to it. And the one learning that we came away with was: Infrastructure costs money.

Paul Hinze: Yeah, I mean, we were just as surprised as you are. It just came out. It came through so clearly in the customer interviews.

Robbie Th'ng: When we spoke to customers, where they think about cost in relation to infrastructure is the part that was a little bit more surprising.

Paul Hinze: Let's speak about where we are with this feature. What we're doing is we're taking that space in between terraform plan and terraform apply. This is something that's unique to Terraform. It's one of its signature features, the ability to predict what it is about to do and present that to you before it takes any action.

What we decided with this feature to do is to answer the question, What if in addition to predicting its operations, it could also predict the cost of those operations? This is one of those things that is distinct about infrastructure.

There are not many organizations that are taking the cost of an application deployment, but cost is absolutely something you're thinking about for infrastructure. And by pulling forward this ability to predict cost to the plan phase before the changes are made, you get some really cool functionality.

Robbie Th'ng: It's hard to understand the implication of cost when you're doing an infrastructure action, and most companies today are probably exploring costs with their bill. They're probably either using cloud-native services built into their cloud provider of choice or they're using a third-party tool.

And that's good. You should definitely keep doing that. You should understand how you're paying for infrastructure. The problem with that is it comes after you've paid, and that's not bad. That's just how billing works. But it's different from the way we've thought about it when building this feature.

What the feature allows you to do is it gives you this cool cost culture around provisioning. And that's nice, because what you start with on this slide is a very high-level breakdown of the delta of change. Like Armon was talking about this morning, this kind of thing that you can like look at and be like, "That doesn't seem right," or maybe, "It seems completely fine."

The next thing you can do is like dig into it and, at the resource level, understand what's impacting your change the most. It can take you to some pretty interesting places if you're looking across multiple clouds, for example, and a lot of situations like that. But it doesn't stop there. It gets more interesting with other parts of the pipeline.

Paul Hinze: The power of taking this predictive estimation of cost and pulling it into the side of change is hard to express alone. We're really excited for you to dig in and see what this does to create a sense of cost consciousness in your daily operations.

You combine that with the Sentinel policy-as-code engine and you get some really powerful use cases that you can express here.

That's exactly what we've done from Day 1. All of this cost estimation data is available and addressable from Sentinel policies. That allows you to express real-world situations. If you can imagine a team going to their manager and saying, "How do we think about costs for this new application?" The manager may say, "I just want you to talk to me if it goes over $1,000 a month." That's a Sentinel soft-fail policy. You can encode that reality into your Sentinel policies.

Maybe you're a team that has a limited budget. You can't spend more than $1,000 a month. You create a Sentinel hard-fail policy for that, and you're going to be prevented from spending that money before the change gets to production. You're going to have that alert before you incur the cost.

Some really powerful things happen when you start to add Sentinel into the mix with cost estimation.

Two takeaways

Robbie Th'ng: Our 2 takeaways from this. First, we really want you to try it. Like Hinze was saying, we'd love you to just give it a shot and let us know what you think. This is just the start of this feature. We can go in a lot of ways with this. So your feedback means a lot.

Just like everything we do in Terraform, we'll do it together. So, what you tell us will inform where we go. Let us know how you're using it. Let us know what features matter to you, like what cost means in your organization, and we can adjust and adapt.

Second, we are in this plan-and-apply phase. We can do a lot there. This does not need to be the only thing that we inject into this part of the run. We can do things before the run and after the run. You might have some ideas based on what you're doing with Terraform about information that would be useful to you in that context.

Paul Hinze: Give them to us.

Robbie Th'ng: Yeah.

Paul Hinze: Give us those ideas.

Robbie Th'ng: We just want to hear. Speak to anyone at HashiCorp related to Terraform. We just want to understand a little bit more about what you could be doing with this kind of feature. So give it a try and let us know.

Paul Hinze: Cost estimation is available on Terraform Cloud today in the governance tier, and it's available in next month's release of Terraform Enterprise. With the initial release, what we have is support for some of the most common resources.

We have what we call price resolvers for those resources. And we're going to continue adding price resolvers in the coming weeks and months.

Give it a try. We can't wait to see what you think.

Terraform Enterprise

Robbie Th'ng: I think now is a good time to talk about Terraform Enterprise. Terraform Enterprise is, quite literally, the product that we're building together. It's reached this massive maturity point, which has been amazing.

We've gotten it to more customers and more industries than I think we ever expected, which has been amazing. And the customers have been phenomenal. They are so vocal about their feedback, about how they're using the product, about what they expect from the product. That part has been awesome. Taking in that feedback has got us to where we want to take the next version of Terraform.

Paul Hinze: It's with these customers that we came to understand this Terraform-as-platform analogy, and it's where we've been building all of these features. And with respect to our Terraform Enterprise customers, the signal we've been getting is the importance of this platform as it becomes woven in to the daily operations of a modern DevOps-practicing enterprise.

If you think of something as a tool, that's something that you can pick up off the shelf, use it, and then put it back on the shelf. But when you're starting to think about this as your infrastructure provisioning pipeline, you start to realize that your business-critical operations are dependent on this platform.

That's something we've been taking to heart of late and something that we've been thinking a lot about: What do we need to do in response to this signal? This is a mission-critical platform, and we need to respond to that.

Robbie Th'ng: When we talk about it being mission-critical, it usually means 4 things based on the customers that we've had a chance to speak to about this:

• Highly available

• Scalable

• Flexible

• Operable

The high-availability one, that's a just a no-brainer. Everyone needs something that's highly available, but it has to be highly available by a configuration that works for your organization. Something that you know, you get to define.

It has to be scalable. For some customers, they're running Terraform Enterprise with a small team inside a large organization, but they're doing all the work. But for others it's thousands of developers all interacting with Terraform, at a lot of different locations. It has to be able to scale to that level of demand.

It has to be flexible. It has to be something that can be installed in the environment that you have. The environments that we test in are often not the environments that are out there. And we have to be able to accommodate more and more configurations just in order to be able to meet that.

And finally, it has to be operable. It has to be easy to install, and it has to be incredibly flexible to manage. We built it with this in mind, that people would install it once, it would just exist as a thing. But that's just not what happened at all.

People install it multiple times for multiple different reasons in different environments. So we have to make it significantly easier. There are other things that end in "able" as well that it has to be, but we only could think of 4 for this.

The new Terraform Enterprise clustering architecture

Paul Hinze: It's with all of this context that we are really excited to walk you through the new Terraform Enterprise clustering architecture. This is the latest in a set of iterations that we've been doing alongside of our customers. It's the culmination of years of engineering work, of years of work alongside some of our biggest customers.

We think it ticks all the boxes on what you need a mission-critical infrastructure provisioning platform to do. Let's dig in and see how it does that.

Robbie Th'ng: The key for the first feature is in the name. It's based around first-class clustering primitive out of the box. It's meant to be deployed in a minimum of 3 nodes, and any 1 node can be knocked down and the cluster is still alive. All 3 nodes are serving traffic, which gives you a much better availability-on-performance profile already. And it doesn't stop there. It doesn't stop at 3.

Paul Hinze: You take this base cluster, of minimum size 3, which gets us into our sibling products territory finally. We're very excited to say you need 3 nodes for Terraform Enterprise, please. Take that, Consul.

You take that base cluster and you get this scaling property as well. You can add more nodes to the cluster. We've successfully launched clusters of up to 100 nodes. If you've got a 101-node cluster, you're going to be exploring new territory with us.

Robbie Th'ng: Just collapses at 101 nodes.

Paul Hinze: He keeps saying it's going to collapse at 101 nodes, and I just don't think it's going to. We'll test that tomorrow.

Robbie Th'ng: OK.

Paul Hinze: What do you get from scale? You get control over the availability profile. You can knock out more nodes and still be serving traffic. And you can get a performance profile as well. You can get more users successfully using the platform.

Robbie Th'ng: Terraform Enterprise needs to store data. That part probably isn't that surprising, and we store data in a data tier that we ask you to set up when you install Terraform Enterprise.

There are 2 kinds of customers generally that we speak to. The first kind is like, "We want to manage that data tier. We want to have the level of redundancy and control over it that we need for our organization. Tell us what the data primitives are. We'll go away and we'll do that."

But other customers often don't want to do that for a variety of very good reasons, whether it be regulatory reasons or compliance reasons. There's a lot of reasons why you might not want to run the data tier primitives that we're asking for.

With this new architecture, you can do a Terraform Enterprise production-grade install without the external data tier.

It can go 2 ways. You can either have the data tier like you have today and just keep running that externally and giving you that control and redundancy, or you can just give it to the cluster and the cluster will manage the data tier for you.

This works with the scaling that Hinze was just talking about. If you scale up, then Terraform Enterprise will scale up the data tier for you as well. All you need to give us our compute nodes now. That's a much better way to go into a lot of customer sites which are restricted in terms of things that they can provision and install.

Paul Hinze: Finally, this architecture comes with the latest and greatest with respect to the supportive tooling that we've learned from customers is important to make this easy to operate. That means easy to install and easy to keep running.

This includes Terraform modules that allow you to install some of the most common configurations of Terraform Enterprise with just a terraform apply, which is pretty cool.

And then, the pre-install checks that allow you to verify that your environment's ready for a successful install, and health checks to help you ensure that the cluster stays running and healthy throughout its life as a happy little cluster.

Finally, a really cool migration tool that'll help you get from prior editions of the architecture into v5.

Robbie Th'ng: That's the next version we've been internally referring to as v5.

Paul Hinze: Yep. It's a mystery as to why we're calling it v5.

Robbie Th'ng: Yeah. Who knows how many iterations there were?

The beta is running now. So, some customers here were in the private beta. Thank you very much for your help in getting us feedback that allowed us to get to this point.

It will be available later for everybody as a public beta and eventually as GA. But if you want to be in the private beta, speak to any of the HashiCorp booths about getting into the Terraform Enterprise private beta, if that's something you're interested in.

The power of Terraform’s providers

Paul Hinze: In closing, we'd like to highlight a very important part of the Terraform ecosystem, and that's providers. Terraform has really enjoyed this very specific and unique success because of its ability to address all of these APIs out there in the world with this infrastructure-as-code declarative model.

Every API that exists requires a provider plugin. And the thing that goes often overlooked is how much effort goes into building and maintaining those providers. It's thousands of person-hours of effort that go into that.

Of course Terraform Cloud, Enterprise, the core of Terraform also have a ton of work. But the work to maintain these providers is really important and it requires the work of a bunch of distinct groups that we wanted to recognize and appreciate.

Robbie Th'ng: It's 3 groups really. The first we want to shout out is the HashiCorp people that are working on providers all the time: the engineers and the product people and the partner people that are always working to make sure that we're working the right things, fixing issues, checking community pull requests, making sure that we've got the right level of coverage with our cloud partners. A huge thank you to those people. Terraform wouldn't be what it is without all of your efforts.

And we can't thank that group obviously without thanking the major cloud partners that we have. We have first-class relationships with every single major cloud provider, which is just an incredibly fortunate position to be in. We are very lucky to have that. And it allows us to do the things that we do. I think it's gotten Terraform to where it is. There are not many other vendors that can say that. It's pretty important to us.

And finally the open-source community. Terraform wouldn't be what it is without the open-source community. Always helping us find things to work on, helping us contribute to the providers, expanding the provider ecosystem as well. Like always adding more providers. Everyone who's involved in any of these groups has helped us get to where we are.

Terraform is its ecosystem. Terraform is it's providers, so a huge thank you to all of you. And thank you to you as well.

Paul Hinze: Thanks to you. You're participating in these feedback loops that produce Terraform, that produce the solutions to these problems, because we discover those problems with you. So thank you for being here. Thank you for coming to our talk, and we're looking forward to continuing to build Terraform together with you.

Robbie Th'ng: Thanks, everybody.