A (2nd) Tour of Terraform 0.12

At HashiConf 2018, Terraform core engineer Kristin Laemmert gave an early tour of Terraform 0.12. Now at the backend of its GA release in May 2019, we're revisiting the tour with the latest syntax changes and an eye toward the future.

Some of the updates covered include:

• Improved error messages
• Rich value types
• for expressions
• Conditional expression improvements
• Machine-readable output
• and more...

Slides

Transcript

Hi. We're here to talk about Terraform 0.12. This is your last chance to decide that you're in the wrong room. Okay, cool. I am Kristin Laemmert. I am a developer on Terraform Core at HashiCorp. I've been here a little over a year. You can find me at GitHub and Twitter as @mildwonkey. I'm also Kristin in Slack. Outside of work, I am mostly known for my ridiculously photogenic cats. There you go, yeah.

We are here to talk about Terraform 0.12. This is a major milestone release. We released it in May of 2019. So I have to start by answering possibly the most common question I've heard about Terraform. What took you so long? Folks were a little annoyed at the wait. Our original estimates had us releasing Terraform in August of 2018, so we missed that by a little bit. Some folks were frustrated like Cheetara here. It just took longer than we expected, and I am sorry for this. Now, I'm not here to make excuses, but I do want to give you a hint of the difficulty of the task we went under.

For Terraform 0.12, we rewrote the underlying configuration language. And while this is a change we tried to weave in nicely, in the end, large chunks of the codebase had to be entirely rewritten. The analogy I like to use is that we had to lift up the house to replace the foundation underneath while there was a dinner party going on inside.

Again, Terraform 0.12. We are here to talk about enhancements to the Terraform registry, the new Terraform configuration language, some new machine-readable outputs. For each subject, I'm going to give you a little bit of background, I'll cover some of the many enhancements, and then I'll tease some of our future possibilities. Now for those of you, like me, that have a really hard time listening to people talk, I have both in Slack and on Twitter tweeted out a link to these slides and my transcript, and in these following slides, I'm using some example code. It's terrible, I don't use this, but you can see it and GitHub if you want to follow along. And I'll give you all a moment to take pictures.

Terraform registry enhancements

Up first is the Terraform registry. We're going to talk about providers in the registry. If you're not familiar with the registry, it is a website where you can publish, discover, and share Terraform modules. These range from official modules that were vetted by HashiCorp and published by our trusted partners, to community modules that anyone can upload and share.

Now in Terraform 0.11 and earlier, when you ran terraform init, Terraform would load up your configuration, get a list of the providers you need in any version constraints, and scrape S3 bucket basically, releases.hashicorp.com, to get a list of all the binaries and versions. Starting in 0.12, Terraform is now clearing the registry for this provider metadata. All right, this is a topic that's more exciting for our future plans.

So, ta-da, great. So right now it's much like modules, you can only get official providers downloaded automatically on Terraforming net from the Terraform registry. But we've laid the foundation so that people will be able to publish their own providers. Not only can you publish your own provider either to use yourself or to share with others, but this means those providers will be downloaded automatically on terraform init. Now for some advanced users, you may already know that it is actually possible to run your own implementation of the module registry. You have to figure out the APIs yourself. And in the future work we're doing, we'll probably make it easy to do the same thing with the provider registry. Of course, if you are a private Terraform enterprise customer, you already have this and you don't need to run your own. All right, that was a quick section. We're going to move right along.

Terraform configuration language

The bulk of this talk is going to be focused on enhancements to the Terraform configuration language. I've got a lot of points to cover, and this is barely scratching the surface. In Terraform 0.11 and earlier, the Terraform configuration language was made up of two main components and some native Terraform bits. These components were HCL, which is the HashiCorp configuration language, and HIL, the HashiCorp interpolation language. Oh, that's unfortunate. Okay. Squished slide. Sorry. So the HashiCorp configuration language is where you get the attribute in block syntax like this variable definition here and a resource block. HIL is what provides the ability to embed functions and variables in strings. So that's where you get that quotation, dollar sign, curly bracket syntax. In 0.11 and earlier, basically everything was a string or embedded in a string. You've written Terraform, you know this.

Terraform 0.12 introduces HCL2. This library merges HCL and HIL into a single library. It introduces a robust type system and it enabled us core developers to really address this huge list of enhancement requests, and a handful of bugs. So we're going to take a look at some of these changes.

First-class expressions

Expressions are native to HCL2. So things like functions and variables can now be used directly outside of string interpolation: ("${}"). You can write many fewer quotation dollar sign curly brackets, and that is even harder to say than it is to type. The other implication is that lists and maps can be used directly. So instead of needing this ugly list function to do an empty list list(""), I can just use brackets. Okay. We're going to look at a few examples. I'm starting with a side in a Terraform 0.11 syntax, and then I'll show you the difference in Terraform 0.12.

And this is up in the GitHub repo. Like I said, some very silly example Terraform code where I'm creating a docker image and then a docker container from that image. Okay. So I'm just going to draw your attention briefly to the section in the middle where I've just, referencing a couple of resource attributes. I've got a name and a shasum.

And now we're going to take a look at this in a 0.12 syntax. Okay. Not a huge difference in a lot of ways, on the outset, this is just an aesthetic change. It's easier to read. I think it's easier to read without all those dollars and quotes and curly brackets, but not a huge change. So let's see if I can show you something more interesting. We're going to go back to a 0.11 example. Okay, in this slide, I'm creating an AWS instance, and I've got this kind of gnarly conditional statement at the bottom.

I have a variable called vpc_security_group_id. What this conditional says is if that variable is an empty string, which is the default, sorry, if it's not empty, pass it to the attribute in brackets because it needs a list. If it is empty, just use the list() function to pass an empty list. I have written actual real functions that look like this to deploy infrastructure. It's confusing and it's hard to read even when you're the person who wrote it.

Let's take a look at this in 0.12. Okay. Much like the previous set of slides, this is mostly an aesthetic change. I find it easier to read, but it's maybe not a huge difference, but I can actually make this look a lot better. And it's a good segue into our next topic. So let's look at a neat thing I can do. Sorry, I missed something to highlight. So in 0.12, I'm going to go to 11, to 12. I can also use those empty brackets instead of the list("") function again.

Okay, here's my fancy new version of the slide. Instead of defaulting my variable to an empty string, I can use null. And now this conditional statement is much easier to read and read as an about. This is a format in syntax anyone who's written code is going to be used to. So null is a new concept in Terraform 0.12 and will lead me into the next section.

Rich value types

In HCL1 Terraform 0.11, as I said before, nearly everything was a string or wrapped in string interpolation ("${}"). We did have numbers and booleans as well, and you may know that booleans would occasionally magically get transformed into a 1 or a 0 when you weren't expecting it.

Also, lists and maps. Well, HCL2 makes a rich type system with native support for numbers, booleans, complex values. It also allows you to use an entire resource or module as a value. So let's look at a couple of examples of that. This first slide shows a variable, which is a complex object. I define my variable with its type. This is a list of objects. Each object has three attributes. There's internal, external protocol. And each of those attributes has their own type. Doing this in HCL2 means that Terraform can validate your input variables when you run terraform init instead of waiting until it gets passed to the provider because I know exactly what type I need these to be.

Another implication of these rich value types was being able to pass an entire resource as a value. So if you've written a Terraform module, you've probably had to add some outputs so people using your module could reference attributes of the resources you've created. This is a really nice feature. The problem is, every time a user comes up with a need for another output or another attribute, you're going to have to refactor your module to add the new output. And this can get tedious, if you have to keep refactoring your code every time someone wants an attribute, that's just a lot of unnecessary code touching. So in Terraform 0.12, I can just decide to output the entire resource.

So fun fact, this particular resource, the docker_container exposes 53 attributes. Even if people using my module only need 10 of those, I've saved myself 30 odd lines of code. So this is efficient. This is powerful, and this lets your users decide what's important to them.

Improved error messages

Now this is where I really want applause. When I gave this talk at HashiConf last year, I was embarrassed to admit how excited I am about improved error messages. But when Paul Hinze mentioned it at the keynote, he got applause. So come on. Error messages. Where's my applause? Okay. Yeah. This is a very real error message that I myself have seen many times in Terraform 0.11. Apparently, there's a syntax error in my main.tf file. That's nice. That file could be thousands of lines long. I have no idea what to do with this information besides maybe cry.

The HCL2 parser retains detailed location information about all syntax elements and tokens. So error messages can refer to your specific source location including file name and line number. It can also show you a snippet of the source code in context with generally a description of the error and frequently a suggested fix. As we can see here, on exactly line 24, I am bad at equating them.

In this example, I've got a type mismatch error and it shows me the function that's a problem where I'm clearly adding a number to a string. And finally, in this example when I've created a typo, Terraform is able to identify that and make a suggestion that maybe I meant output instead of outptu. Now, improved error messages are the feature I'm most excited about because frankly, I spend a lot of time writing and running Terraform code to help debug problems, and these error messages are more useful than anything, any other tool I have as a core developer.

"for" expressions

However, I think I know what most of you are going to be more excited about, and that's Terraforms for expressions. Now it's not quite loops, but we totally have for loops. 0.12 introduces a concept called for. This is used to transform lists or maps into other lists or maps where you can transform and filter the elements. We also have for_each, which is used in conjunction with dynamic to create a collection of nested blocks dynamically. And for_each is it's on its own line because we have some really cool future plans for this one. So this is where I expect you all are going to be most excited.

Okay. We're going to start with the Terraform console. If you were not familiar with this command, I highly recommend it. I didn't know it existed until I started working at HashiCorp. The Terraform console command when you run it, will load up your current configuration and state file and let you do things like play around with functions, and test out your code before you commit it. In this example, I have a variable that's not shown called, list. That's that var.list. It's just got three lower case letters, a, b, c. In the first for expression here, I'm creating a map of the lowercase version of my list to the upper case version of the element in the list. This is very sad. Okay, we'll skip this so you can actually read it quickly.

In the second example, what I'm showing is that you can use conditional statements to filter your list. So here I'm creating a list of the upper case versions of the elements as long as it's not B. Okay, well, this is about as contrived as an example as I can come up with, so we'll see a better one in a couple of slides, and we'll just, there we go. Okay. If you're unfamiliar with what I'm saying when I say, dynamic nested blocks or nested blocks, in this resource here, the docker_container accepts one or more ports blocks, and each of those blocks identifies a port that's being exported on the container. You can end up having to write a lot of these blocks that can go on forever and ever.

So in Terraform 0.12, we can use the dynamic construct to create a single ports block. And for each, to identify, in this case, what variable I'm iterating over to dynamically create my nested ports blocks. In the example I'm using in the code in GitHub, I think there's something like 10 ports I'm exposing. So has saved me 58 lines of code to be able to do this in a single block. There we go. dynamic "ports" and for_each. Okay. I promised you a more realistic example of what I can do with a for expression. As an infrastructure engineer at my previous position, I spent a surprising amount of time formatting outputs to look just the way I wanted them to. This makes it a lot simpler. I'm just using a for expression to get the outputs just the way I like. I don't want to admit how many times I had to copy and paste outputs or pipe into the file. So now I can get it just right.

Everything else

Okay. So we covered a handful of topics, but there's so much more that I don't have time to talk about even though I'm a super-fast talker when I'm nervous. So what I'd recommend if you're interested in all of the updates in Terraform 0.12, is that you go to hashicorp.com and take a look at the blog posts we released leading up to 0.12. There's something like a dozen posts up there covering all sorts of new enhancements. The changelog is also really interesting. And of course, I promised you some future plans.

Future plans

Now there was a lot of features that we really wanted to get in the 0.12.0 release. But as I said earlier, we ran over on time. So some of these I can't really promise when they're coming, but these are things we want to work on.

• depends_on for modules - Module count - Module and resource for_each

I hope you all are just this excited to rush back to your desks and go start refactoring your code, or take a nap. Again, this slide is not just to show off my cats. I need to stop and breathe.

New machine-readable outputs

We're going to take a look at two Terraform CLI commands as well as a new tool that we've open-sourced. And of course, again, look at some future possibilities. But why do I have a whole section on machine-readable outputs? Honestly, this isn't something I thought that Terraform users were really going to care about, but you do care. There are many users doing really cool things with the output of a Terraform state or plan. And there's a really healthy ecosystem of third-party tools that are parsing terraform plan and state file data.

This is really not designed to be parsed by anything other than Terraform itself. So we've designed some machine-readable outputs to help with that. I want to make it clear that I'm not trying to suggest that these tools aren't necessary anymore, that we're replacing them. The Terraform Core team wants to help you do what you need to do with our data. So our goal with all of these outputs is to make compatibility promises so you can confidently write tools that read, plan, and state, and config data knowing that it can work with future versions of Terraform.

Show command

The first command we're going to look at isterraform show. If you're not familiar with the command, its default usage will just print out this impossible-to-read file of everything in your Terraform statefile. I don't expect you to be able to read anything here. This is just to give you an idea of the shape of it, we've added a new flag to terraform show: -json, here I'm piping it through jq because it doesn't pretty print. It is impossible to read again. But as you can see, I have my Terraform state data in a JSON format. You can also see, no, you probably can't see. But at the top there is a format version at 0.1.

So as I said, we hope to make compatibility promises. If we're introducing breaking change to this format, we'll increment that to 1.0. And that way you can write tools that will work with the various state files or be able to detect if we've broken something. All right, so again, this is too much information, you can't read it, but you can do a lot of cool things of course, once you have your state format in JSON.

So in this example, I'm just writing a little query to get the addresses of all of the resources in all of the child modules. And of course, you can do whatever you can think of with the state version. You can also use the terraform show command to read a planfile. So in this example, I'm running terraform plan, I'm writing my plan out to a file, and then I'm using Terraform to show JSON to read the plan file, and querying to get a list of resources and their plan changes.

Ta-da, resources and their plan changes. Okay, so, and again, you can do whatever you can imagine. This is everything that you see when you're on terraform plan, and now in a machine-readable format. There's more in a planfile than you might be aware of. A terraform plan file includes a snapshot of your current state, and all of your local configuration files. So when I'm querying this plan.json, I can do things like read my configuration, and get a list of all the modules and their variables. And here in this example, I'm including descriptions because it looked nice.

terraform-config-inspect

Now, if what you're interested in is inspecting Terraform configuration, you'll be interested in terraform-config-inspect. This is a third-party library that HashiCorp has open-sourced recently. terraform-config-inspect was originally designed for the Terraform registry. We didn't want the registry to need to be tightly coupled to the version of Terraform, and so it needs to be able to parse modules that are written in both 0.11 and 0.12 syntax. So we wrote this library so that it could parse modules and both syntaxes, and then we added the CLI so users can do the same. And here's just a default usage where I'm running terraform-config-inspect to get this JSON output of all of my Terraform configuration files.

Providers schema command

The last command I'm going to show you is terraform provider schemas. This is a very specialized command. What this command does is parse your configuration files, get the providers, and print the full schema for all of the providers. This is a lot of information. In my example code, I've got three providers, docker, null, and random. These three providers print out nearly 2000 lines of code. It's hard to read, but it is something that a number of users have asked us for. So here I'm parsing the provider's file to get a list of all of the attributes of the docker_container resource. And as always, you can pretty much do whatever you want to with this information. The power's in your hands.

Honestly, I don't have a lot of future plans for the machine-readable outputs yet because I need to see how you use these and what your requests are. But I mean, there's some potential ideas. Maybe we'll add YAML as an output. Probably not. I don't think we're going to use the YAML.

There's a lot of documentation available for these machine-readable outputs. They get long and involved, so I hope you check these out and you can also find the repository where terraform-configure-inspect is:

• terraform show-json - Terraform JSON format - terraform-providers-schema - terraform-config-inspect

We have many resources available to help you upgrade from Terraform 0.11 to 0.12. There is an upgrade command. If you don't know it, it will rewrite your configuration, make all the changes it can, and print a little to do note if it needs a human to figure out how to fix something. There's an upgrade guide on the website, and I've said before, the changelog is a really excellent resource if you'd like to get into the nitty gritty details about what's changed. We also have some general resources to help you learn, and I hope you're inspired now to share and contribute back to Terraform either by writing code and making pull requests or even just opening bugs and feature requests. We need to hear from you to know what you like and what you don't like, what works and what doesn't work. We depend on our community.

Mitchell said, thanks, better than I ever could. The 0.12 release was a huge undertaking. A lot of work, and many teams helped us with this, so I want to at least call them out. I'm not going to read this to you, but besides the core team who was the obvious ones, there's also the ecosystem team. Our Terraform registry team, Terraform Enterprise, the education team, and then honestly, everyone else. So many people both at HashiCorp and in the community by downloading the Alpha and Beta and helping us test it, helped us get Terraform 0.12 out the door, and I'm personally grateful. I want to thank you. I want to thank HashiCorp. I want to thank everyone who helped. And on that note, I'm due for a vacation. Thank you.