A Guide to Terraform Binary, Provider, and Module Versioning

Simplify and automate version management for Terraform binaries with tools like semantic versioning, tfenv, tfswitch, tfmigrate, and the compatibility guarantee of Terraform 1.x.

»Transcript

Brian Menges: 

Welcome. Today we're going to talk about versions and all the versioning bits and pieces that are available to you with Terraform. There's the Terraform binary itself, and then you have the providers that Terraform provides, and also the modules. 

We're going to journey from the very beginning to the end and take you along all the versioning bits and the upgrades. I'm Brian Menges. You can find me on Twitter, LinkedIn, and GitHub, last name and first initial. 

You're always welcome to contact me and reach out. I will help where I can, and there are a couple of public community Slacks that I'm on, like Google's and such. 

With me is Nic Jackson.

Nic Jackson: Hi, I'm Nic Jackson. I'm a developer advocate at HashiCorp. You can find me across the internet in a bunch of different places: on the HashiCorp forum, the Cloud Native Computing Foundation Slack, Twitter, LinkedIn, GitHub, and I'm sure you can find me somewhere. Anywhere.

Brian Menges: I work for Anaplan, so it would be wise of me to welcome you to come join me and help us build great things. Here at Anaplan, we do corporate and business planning. We would love to grow our engineering organization and bring some better products and innovate in the industry, so take a look at the careers link and you can find open engineering jobs and other areas.

»Semantic Versioning

At the beginning of our journey, we need to talk about SemVer, the Semantic Versioning Specification. That has a couple of constraints; you have the major, the minor, and the patch, and they have the syntax that comes along with that. With the syntax, you have both exclusive syntax and inclusive syntax. Exclusive means that the version that you indicate is not included by virtue of it being excluded. Inclusive is: I want to begin with this version and include it and either go greater than, less than, or be exactly equal.

What's special is minimum specification, which is ~>. ~> allows you flexibility with your versioning. It allows you to say, "0.12.0," and when you use ~> that allows you to have all 0.12.x patch versions. 

If there is an update to that patch, because you've specified 3 segments, you can get anything from 0.12.0 on that patch all the way forward. This works for every segment. If you do 0.12, that'll include all 0.12 and higher, so 0.13, 0.14, and so on. We'll get further into that and you'll see that theme constantly present.

»The History of Terraform

We talked about Terraform versions and the binary, but let's recount a little bit of the history. I could think of no better person than Nic to recount that for you.

Nic Jackson: Cheers, Brian. Let's go on an archeological tour of Terraform from the very big beginning, in July 2014, to this summer, when 1.0 came out.

At the beginning, Terraform was released 0.1, July 2014. It was released with a limited number of providers, AWS and DigitalOcean. Predominantly, the idea was to show how the system worked, how you could use Terraform, and to enable people to start building into the ecosystem. 

Now, I'm going to skip a few years, because at the beginning of Terraform, it was a lot of provider development, it was a lot of features, and interpolation functions, and things like that. The real important changes started to happen when we started to get stability, you could say, but more a sort of commonality in the use. It became more popular.

»The Dawn of Data Sources

In 0.7, we introduced data sources. Data sources were a major change in the way Terraform worked, because a data source allows you to read the properties of a piece of infrastructure, as well as being able to create them with your resources. This allows you to build richer and more dynamic Terraform configurations.

Shortly after that, at the end of 2016, we released 0.8. Brian's going to talk about this, but 0.8 was when the Terraform block was released and when the version parameter was introduced. This is when you could start to say, "This Terraform configuration is associated with a specific version of Terraform." That was really needed because we were adding a lot of features and changing the system very quickly.

Terraform 0.9 came around the middle of 2017. It was an important release, because it brought state locking, the ability to collaborate on Terraform configuration. You had the ability to store Terraform state remotely in things like S3 buckets, but now you had the ability to lock that state. It allowed multiple people to collaborate on the same Terraform workflow. Previously, this was only possible if you were using Terraform Enterprise or some third party like Terragrunt.

»Growth of Providers

Terraform 0.10 was a really important release, because we started to distribute the binaries for the providers independently of Terraform. That was important because Terraform was growing at an astonishing rate. I think around this time we were running around 70 providers. In 3.5 years, we had added 70 providers. 

When we were distributing the provider as part of the Terraform binary, if a change was made to, say, the AWS provider, you had to release Terraform. That's not sustainable, and it didn't create a good community workflow either, so we started to break this down. The binaries became independent. The providers could have their own release cadence, and Terraform could have its own release cadence.

In 0.11, announced at HashiConf in Austin, Texas, I believe, we released the module registry. You had been able to build modules for a while, but we wanted a way to enable the community to easily share those, to be able to distribute those, and importantly, as we'll see later, version modules. Terraform 0.11 was the step toward something that a lot of people in the community say was a big change: 0.12.

»Terraform Gets HCL2

Terraform 0.12 was the introduction of HCL2. In the summer of 2018, we started to talk about 0.12. We released a series of preview blogs where we talked about the new language features, why the language features were needed in order to be able to push Terraform forward and to be able to get to where we can say, "It's now language-complete or near language-complete, and there'll be no more breaking changes." But it was a lot of work. That blog post stated, "We're going to release Terraform 0.12 with HCL2, very big language changes, in Q1 2019.

The work for HCL2 started about 6 months prior to that blog post. By the time we released it, which was around summer 2019, there'd been about a 2-year period of working around HCL2, the new interpolation features, the new language features, and some of the dynamic blocks that the community had been asking for. 

It was a huge undertaking by the Terraform team and it was a huge undertaking by the community, as well. There was an introduction of changes, which you had a change of configuration for.

Terraform 0.12.18 was a minor release, where we were still iterating, but in 0.12.18 we started releasing binaries that are signed for Mac platforms. We were making sure that the Terraform environment was working in the way that it should be on the developers' computers.

Terraform 0.13 was a big release. 0.13 came at the beginning of 2020. Now, with 0.13, we took that provider ecosystem and we did what we did with the modules, with the registry. We enabled providers that were written by the community, to be distributed and to be socialized through the Terraform registry. You can now find providers for various different solutions and tools out there, and Terraform can download those, and it can import those automatically as a separate binary.

With 0.14,, we're still evolving and marching toward the1.0 release iteratively, little by little. 0.14 introduced a big change around state. We introduced code into Terraform that enables it to read different state files. 

Previously, this was a bit of a sticking point, but from 0.14, Terraform could read a huge number of state files, which gave it more backward compatibility and more forward compatibility as well. Where we were going with this—and the state file hasn't necessarily changed, it's still on version 4—but we wanted to get that 1.0, that compatibility promise.

Terraform 0.15 introduced some more breaking changes. We removed the quoted variable constraint, so if you are defining a variable type of string, you don't put quotes around it. We changed some of the interpolation functions, so list and map became tolist and tomap. These, we hoped, would be the last breaking changes in the language before we'd get to 1.0.

»1.0 Arrives, with a Promise: No More Breaking Changes

This summer, 1.0. We finally got there. The thing about 1.0 for us, it wasn't about production-readiness. Terraform's been production-ready and used in production since that original version back in 2014. It was about a compatibility promise. 

With version 1.0, we were making the statement to the community that we are not going to make any more breaking changes to the core language. We aren't going to change any of the core workflow features, things like apply and plan. Core workflow features like that will remain the same throughout the 1.0 lifecycle. 

If you're using 1.10, it will work just fine with 1.0 configuration. If you've written CI/CDs set up for version 1.0, that should work fine with 1.10. That compatibility promise we're going to make throughout the 1.0 release, approximately 18 months, we believe. But who knows what's going to come with Terraform 2.0?

»Terraform Versioning

Brian's going to show us how you can use that Terraform version and dig into that in more depth.

Brian Menges: Thank you, Nic, for the long journey down the yellow brick road of Terraform, and all of its many stops along the way, and the massive development that's incurred over the last 7 years. Let's get a little into some of the specifics.

The Terraform state file is predominantly where Terraform versioning was introduced. It tells you what version of that binary generated all this content. On screen is a small snippet of that, and it'll tell you exactly what that binary was. I grabbed a bit older state file, because there have been a number of different versions. We're up to version 4 now.

Over 7 years of development, that's a very good track record as far as version incrementation and stability. Naturally, with the advent of Terraform 1.0, now you have the ability of 1.(minor version) to be able to support all the state files that are generated for that version for those binaries. If you do run into a situation where 1.1 and 1.2 are operating on the same state file, that compatibility promise now is there for you, whereas previously it was not.

For the Terraform block, this came in at about 0.11.4. This allowed you, in the code, to directly tell the person who's running your plan or your module what your minimum version spec or what your version spec requirement is for this sort of code. You can actually see it in the code instead of going into the state file and trolling through and trying to find what was compatible with it and whatnot.

With Terraform versioning in the binary, you have a couple of options in order to maintain your plan and working and running and operating. One of those options is tfenv, which uses a .terraform-version file. You can see some syntax on screen that allows you some leeway in your compatible versions. When a user comes in, they can do `tfenv, get the version that is for your plan even without having to look at the code, and start to run right away.

Another option that I've used is tfswitch, which tries to intelligently read your code and grab that Terraform block and infer what kind of compatible version you need. It's much better, in my opinion, to try and read your code, but in the case of Terraform Cloud or Terraform Enterprise, those workspaces explicitly cover or are configured for your version that you're going to run or execute. Those will be exact versions. Of course, you're in much better shape if you've migrated up to Terraform 1.0 in order to run a multitude of different versions.

Let's take a look at the example in syntax. Before version 0.13, you could configure the version directly in the provider. In fact, this was the only way before version 0.12 that you had a version meta-argument and that SemVer syntax that allowed you to specify which version of that provider you had.

With 0.12, you have the Terraform block. Well, the Terraform block has been there for a little while, but now we have required providers, and we have a key-value pairing between those 2. You have the provider that you're going to initiate and that version. 

Now, you no longer need that version matching into your provider's configuration, as in the top example on the screen. 

»Namespacing the Providers

When we got to 0.13 and beyond, we discovered a need to namespace that provider, because the community needed a way to push their providers into the ecosystem, and HashiCorp, of course, had to provide theirs. 

For the ones that HashiCorp maintains, you'll see the default or assumed namespace of HashiCorp. Then, for any other provider, say, for Rundeck, you'll see rundeck/rundeck, because Rundeck the provider team provides the provider for Rundeck. Then you still had that version nomenclature, but now we had to break that out between source and version.

What was interesting is that while some of the syntax was still supported and introduced in 0.12.26, the source namespace was always assumed to be HashiCorp. It wasn't until 0.13.0 that we got to see other namespaces exist and other community providers start to interject their custom provider for all their things. 

Now the ecosystem is just absolutely massive in the number of things that you can interact with and control with Terraform code.

Supporting these you, of course, have to maintain a watch, or you do want to keep up to date. There's the Terraform registry, and also there's an API. If you want to take control of things and write things like webhooks or whatnot, you can check with the registry's API and say, "My code depends on these sorts of things," and your CI/CD pipeline can integrate that directly.

You can follow the changelogs at their public GitHub publishing locations. For instance, AWS, Google, and many of the providers, especially the much larger ones, are on github.com. You can track their changelog or be notified on their releases. 

There's also Dependabot, which is still beta support for Terraform. It tries to read your code and infer what kind of things you can benefit from. If your code says, "I want to use the AWS provider 3.54.0," and there's an updated release and they go to 3.55, then Dependabot will automatically open a pull request for your GitHub repo and say, "I found a new version. Would you like to accept this PR?" You can track it and keep it automatic and up to date.

This can be important, but I'd like to let Nic explain in more detail as to why that's important.

»Keeping Providers Up to Date

Nic Jackson: It's incredibly important to keep your providers up to date. With the HashiCorp compatibility promise in 1.0, we want to say there will be no changes to the language, but the providers are iterating separately. 

There are a number of reasons that a provider might have to change. It could be a new resource that is added. It could be deprecated parameters in a resource. It could be bug fixes. It could be the deprecation of a resource itself, or it could be a change of an underlying API.

Let's say you've got something that doesn't change very often, your core platform. Maybe you only touch it every couple of months. You want to still be iterating over that configuration and making sure that the provider is as close to edge as you're comfortable with. 

The reason you want to do that is because you want to know that that configuration still works and whether there are any changes needed, if you need to change a parameter or something like that in a resource, you want to do it before it's urgent to do so. 

What I mean is that, when the phone rings in the middle of the night, because something's gone horribly wrong and you need to rebuild your infrastructure, the last thing you want to be doing is dealing with updating Terraform configuration or any infrastructure as code configuration at that point in time. 

Get into a good practice. You could have a nightly plan that does a sanity check, or you could have a calendar entry to keep it up to date, or use some of the tools that Brian's talked about, such as Dependabot. 

Try and keep it as close to edge as possible. Make sure that you're always running a provider that's compatible with the cloud APIs and things like that, so that when you need to make those changes you don't have any problems.

Brian Menges: One of the feature requests that I've asked for is to be notified from Terraform's registry site, registry.terraform, so that you can see changes there. Just like you can follow on GitHub, we've requested the feature up there, and we hope that'll be accepted soon.

»Module Versioning

Let's bring it up to the end of the road here and go through module versioning. Modules are going to be largely your code, and so it's about the user's impression of that version. Let's recount and go back to our timeline, Nic.

Nic Jackson: Modules are an incredibly important feature of Terraform, one of the things that I genuinely love. I think it was 0.4 or something like that where modules were initially released. It was very early on in the development of Terraform as a tool.

A module allows you to encapsulate something like a VPC, which contains multiple resources in a single unit. Of course, you can go bigger or smaller; that's entirely up to you. 

In terms of how modules have progressed throughout Terraform's history, there were a couple of big events. I think the biggest event was probably the module registry. Versioning modules prior to the module registry was possible. You could add, for example, if you're running GitHub or Git, you could add a Raft tag, and the Raft could be a tag, it could be a commit hash, or something like that.

With the module registry, we added a specific version tag. The version tag allowed you to employ some of the SemVer constraints that Brian talked about earlier, the ability to say, "I want greater than this version or less than that version."That's a big step forward in the flexibility of how you deal with modules. 

I think that the most important part of modules was the introduction of count. I think there are many people in the community who will agree with me that count is something that you've all wanted in modules for a little while.

I've been at HashiCorp for about 4.5 years now, and on my first day, I sat down with my boss at the time, which was Armon Dadger. We were talking about Terraform. I said, "When are we going to get count on modules?" This is way back in 2017. Armon said, "We're working on that, and we hope we'll have that later this summer." He didn't say which year, he just said, "this summer." 

It didn't turn up for another 4 years, but there was a good reason. I joke about it, but the amount of engineering effort that went into being able to deliver count on modules in 0.13 was huge. We had to get through that 2-year piece of development around HCL2, and then we had a couple of minor iterations, and then another version after that. 

Count is here now. Count, and modules, and some of the other parameters, like the ability to specify which specific provider a module uses. They're all entrenched now in that 1.0 compatibility promise. I think, Brian, you're going to show us how some of this works.

Brian Menges: Let's take a look at some syntax in order to get modules implemented. With the Terraform registry, and also if you're on Terraform Cloud or Terraform Enterprise and you have the availability of your own private registry, you specify that source and also that version that we refer to. 

You can see on screen an exact version, with that double equal. In the absence of that, we're assuming that this is the exact version that we're going to match. Of course, we could still SemVer this. 

What you can see here is that we're organized a bit differently. We have this nomenclature of an organization, and then the module name, and the provider that it is for. In this particular case, an organization being terraform-aws-modules or in the case of your private registry, if you want your own, your company or your org identity, so app.terraform.io/mycompany/eks/aws, where EKS is the module and AWS is the primary provider identified for this item.

With a private registry, app.terraform.io would be generally the default location. If you're running Terraform Enterprise, that would be within your own organization, so you'll have your own custom URL. You could still do in-repo or in-path modules. 

Historically, you would do ./ as a reference to where it is that I am currently, and then most people will put them into a directory like "Modules" and then their own individual directory "EKS," for the EKS module in this example.

You also have other interpolations. You have path.modules or path.root. These interpolations are still available to you in order to get to that. Then, of course, for module.eks under a GitHub pull, as referred to earlier, it's highly recommended that you reference some kind of a tag or branch. 

I prefer the tagging method, so you can see there in a reference v17, etc. That'll pull that exact version when it does that URL pull, or you can do a Git SSH pull. 

There are many other methods for you to pull modules, so I implore you to take a look at the documentation. There's a myriad of ways to get your modules saved from things like Artifactory, etc.

»What Makes a Module a Module?

Overall, what makes a module a module? A module is nothing but a stamped version of Terraform code, and what you would determine as an actual module is a full talk in itself. 

Generally speaking, a module is when you need to group a number of resources in a very explicit way or you need to limit the myriad of options into some very specific use cases. Then you're telling your users this is the only way you can use this.

I'll also remind you that you definitely want to responsibly version your items. Hearkening back to SemVer, you want to make sure that if you're making breaking changes, you wait for the major version changes, your 1.x.x. 

If you're adding abilities but not really changing the majority of your code or simply adding features that would be or a minor version change, then patches, of course, are for butt fixes when you need to do things like fix docs or a syntax error. That's when your patches would come in, which would be your last ternary there.

When breakages must occur with those major version changes, you have a couple of options. You have terraform state mv, which is your CLI, In order to move a resource from one particular name to another. And terraform state rm is to completely obliterate state from your items. 

A neat tool that I stumbled across thanks to Nic is tfmigrate, which allows you to codify migrations, so you can tell your user, "If you're using my module 2.0 and you're coming from 1.x, run this script first, and you'll require tfmigrate, and it will automatically handle all of your conversions and remove the things that we no longer need, will move the things that we need, and will inject the stuff that we need." It's a good way to do that. 

Last of all is Dependabot, which can also help you out with your modules and pull requests to get into that.

I'd like to thank you for the journey down the yellow brick road. This has been a fantastic event. Thank you.

Nic Jackson: Thanks very much. Thanks, all, for watching.