A Tour of Terraform 0.12

Join Kristin Laemmert, a core Terraform engineer, for a tour of Terraform 0.12; a major milestone release which introduces some fantastic new features and a few breaking changes. As the latest release of Terraform, 0.12 sets the foundation for exciting future enhancements.

Terraform 0.12 focuses on major HCL (HashiCorp Configuration Language, the syntax of Terraform configurations) improvements, improved error messages, and Terraform Providers in the Registry. The improvements to HCL include "for" expressions, conditional expression improvements, nullable arguments, an exact 1:1 mapping with JSON, and much more!

Many of these changes have been requested for years, and in addition to the immediate benefits, the new HCL engine gives you much more flexibility to introduce new features in the future. Whether you are brand new to Terraform, a battle-hardened practitioner, an interested decision maker, or someone who likes talks with lots of cute cat pictures, this talk has something for you!

Transcript

Hi. My name is Kristin and I'm a software developer at HashiCorp, I work on Terraform. I am here to talk to you about Terraform 0.12. Now I'm sorry to say that text files and CLI tools, are not actually all that scenic. I'm going to do my best.

First I want to tell you guys a true story. Once upon a time, there was a young software engineer. She along with her team of very smart engineers wrote an awful lot of Terraform configuration files to provision AWS resources. These files started out simply, a few dozen lines, maybe two or three files total, but they grew in complexity over time, like things do. Before she knew it, this engineer had dozens of files, scores of resources, tens of modules and it was a big, complicated mess.

That story is actually kind of long, I'm going to skip to the abridged version. What I'd like to share with you is an example of some Terraform code and a real problem that I've run into. Spoiler: I am that engineer.

I'm going to start out with a super-simple Terraform configuration, I want to set the stage. This is my default behavior. Interesting complications will come in a few slides.

So, in this file, I'm using the random provider to create a random pet. This is just a nice, human-readable string. I've got account variable, I've got a default prefix to my string, this is Linus, my cat's name, and then I've just got that random pet using those variables. Nothing exciting, going to run Terraform apply and ta-dah. linus-excited-goldfish. These are really fun, you should play with random pet, this is just sprinkled with awesome, silly names.

Well that's boring, so let's look at a more complicated example. In this next file, I've added a few variables to introduce some optional features, so my default behavior is going to be the same. I just want a single pet name starting with Linus, but what I've done is added this zoo-enabled variable. This is defaulting to 0, and yes that is a string zero, I'll talk about that in a moment. And then a prefix_list that defaults to an empty string, and now in my resource I've updated the prefix conditional and the intent here is that if zoo_enabled is that zero string, just use the default prefix, same default behavior. However if zoo-enabled is anything other than zero, choose an element from the prefix list provided.

Nice, simple, not all that exciting. I can show you an example using these variables, so I set zoo-enabled to one, have a prefix list, have a count, terraform apply and there we go. Now I get a whole list of pet names. cheetarah-loyal-drake is my favorite one here. Okay, so these are my three cat names by the way, Linus, Cheetarah and Li-Shou. You're going to need to know this, there's a quiz.

Now, I owe you an explanation for why I used a string 0 in a variable that probably sounds like it deserves a Boolean or an integer. In this example configuration, if I were to use a Boolean or an int, and say my zoo_enabled is true or false, things would work exactly the way I expect. However, I am not the only engineer who works on this codebase. I know that there are a few places where Terraform can be kind of weird about Booleans and maybe there's a function where you don't expect it's going to translate that into string, so suddenly your true becomes a 1 and you don't expect it, so it gets a little weird, so just for safety's sake, so that somebody else can come in and refactor my code, I'm going to go with a string compare because it's foolproof.

Now I'd like to go back to my default behavior. I'm going to remove terraform.tfvars, I should be able to run terraform apply again, and just get my single random pet name.

Oh no, I got an error, I totally didn't expect that.

What's going on? Terraform says that element may not be used with an empty list, and then it just prints out that conditional again, so this is weird. It turns out that Terraform wants to evaluate both sides of my conditional statement, even though I assume that it's only going to evaluate that first half, since zoo_enabled is 0, so that's an annoying little Terraform quirk that you've probably run into before.

Well, luckily we're all good at Google search, and I was able to find a way to work around the situation, so let's take a look at my prefix conditional again, which at this point is pretty much impossible to read and I'm the one who wrote it, so we're going to zoom in on this a little bit, and what I've had to do is add this concat() function, where I'm concatenating my prefix list, which is an empty list, with another list which is an empty list that has a single empty string in it, and now Terraform can evaluate both halves of the conditional.

And just to prove it, I can run Terraform again, get my default behavior back. linus-driving-giraffe—that's not the best one I've seen. That's weird and that's ugly.

I'm going to let you all take a moment to meet Linus, my default prefix.

Story time's over. Let's get back into the talk.

Terraform 0.12

Terraform was released in 2014, and it quickly became the tool for defining infrastructure as code. The Terraform configuration language provides a high-level syntax for describing how cloud resources and services should be created, provisioned and combined.

Terraform is pretty damn awesome as it was, if I do say myself. But what was even more awesome is the way that our users used it. You came up with really creative and complicated ways of taking advantage of Terraform's features, things that stretched it beyond what it was designed for, things that we couldn't anticipate. And in that, our users started coming up on edge cases, weird little bugs, quirks with the languages like the ones I just described. You found hackarounds and really interesting ways to solve these problems, or least kind of swab duct tape over them, and we saw that. We saw what you were doing, loved what you were doing and we knew we needed to support our community.

To do that, the Terraform configuration language had to grow. The problem is that a lot of the features and edge cases we needed to fix, we couldn't really do without a massive refactor of the fundamental configuration language. For the past year, Terraform Core has been working exactly on that. I cannot exaggerate how hard this project has been. The Terraform Core team had to jack up the house and replace the foundation from underneath it while people were still living inside, and there was a swanky dinner party. I'm not exaggerating about the length of time, I've only been at HashiCorp about five months, this had been in progress for a good seven months before I came on board.

Okay, so I keep talking about the Terraform configuration language, so I'm going to do a quick review. In versions of Terraform, versions 0.11 and prior, the Terraform configuration was up of three major components. HCL, HIL and some Terraform-specific variables and functions.

• HCL is the HashiCorp Configuration Language. It gives us our variable syntax, block structure, like the example you see here.

• HIL is the HashiCorp Interpolation Language, and that gives us string interpolation and the ability to embed functions in strings, like the format function here.

• And then the Terraform bits are just some Terraform bits.

Terraform 0.12 introduces HCL2—this is a complete re-write of HCL that actually merges HCL and HIL into a single library. HCL2 has a very robust type system and it allows us to address our really large lists of enhancements and feature requests that we've had over the years. Okay, you've heard by now referenced, the Terraform 0.12 blog posts we have up. There's no way I'm going to be able to cover all of the enhancements in 0.12 today. I couldn't even fit all of the blog posts into a single slide, so I highly recommend you go to hashicorp.com and read those if you haven't already.

Now we get to get into the fun part. What I'm going to be doing is walking your through a curated list of the enhancements I'm most excited about in 0.12. Up first is…

First class expressions

In HCL2, expressions are native to the syntax and can be used directly, and what I mean by this is you don't need to use that quotation, dollar sign, curly bracket syntax around everything and lists and maps can also be used directly in expressions. Okay, what the heck does that mean?

Here's an example configuration written in Terraform 0.11 syntax. Nothing too exciting, I'm creating an AWS instance, and let's see, we're going to take a real quick look, I've got a list variable for security group IDs and then I'm passing that list into the instance when I create it.

Let's see what this looks like in a 0.12 syntax. A few changes. First of all, list is a native part of the syntax, it doesn't need to be wrapped in quotes. Also you may see that I'm declaring a list of type string, we're going to get to that rich value type in the next section, okay? And then in my resource, you'll notice no more quotation, dollar sign, curly brackets, so that's hard to say in the right order.

This is kind of just a cosmetic change, I mean the code is easier to read but maybe it's not really thrilling so let's look at another example. This is effectively the same thing I just showed you, except now I have the security group ID as a string, a single item, and then in my resource, what I'm saying is if security group ID isn't empty, pass it into the resource in brackets as a list, else just pass an empty list.

Take a look at that in 0.12. Same thing here, I've got a type string no longer quoted, but my conditional just got a lot simpler to read, okay? If security group ID is not empty, pass that, else woo, empty list. On the outside a little change. If you've written piles of Terraform code this is glorious.

Rich value types

Terraform had support for some basic value types besides just strings. We has some lists and maps, but HCL2 greatly enhances the type system. You can do really complicated and interesting things—things you've always wanted to do in Terraform—thanks to this. You can have complex values, you could have a map of a list of a map of a list and you can also pass entire resources and modules as values.

In this example I'm using a module, so I just wanted to give you a quick overview of the kind of theoretical directory layout where I have a main.tf and some Terraform vars, and then a module subdirectory. Here's my Terraform 0.11 main.tf: I've got a networks variable, I'm using that subnets module and then I've got an output. Now for this to be valid Terraform code in the module I do actually need to be outputting the AWS vpc_id, so let's take a look at that module. This is my subnets.tf module, okay? And you can see I've got my output "vpc_id", that's how main.tf could access that.

Well, what if your main.tf also needs the VPC region or any other attribute? Well, you have to go into your module and write that attribute in, so this is this daily refactor if you need another attribute from what you're creating in your module.

Let's see what that looks like in 0.12. First I would like to give you a moment to bask in this gloriously complicated new variable. I'm defining a map of mixed types and this is just a thing you can do. I feel like I should pause here, I see cameras and photos. Okay, now I'm going to show you, just so you can see what it looks like to define this variable, a .tfvars file with my network variable, so that's a nice beautiful map with integers and strings. It's so cool.

Now we'll take a look at that subnets.tf module, looks pretty much the same as the 0.11 example… but wait. My output is the entire AWS VPC, so now I can access whatever attribute I want to from my main.tf. I see heads nodding, like, “Yes, this is awesome,” how much re-factoring do I save you?

There we go, come on. That was what I was looking for.

Improved error messages

Well I was going to share my secret disproportionate excitement for improved error messages, but then I was at the keynote yesterday and you all clapped about improved error messages, so I guess I know how excited we all are.

This is a Terraform 0.11 error message. My main.tf could be thousands of lines long and maybe my editor's kind of crappy, I don't know what closing brace I'm missing. This is not helpful.

Here's a Terraform 0.12 error message. Do I even need to talk or should I just show the rest to you? So, HCL2 parser retains detailed information about the location of any syntax elements or tokens, so not only am I getting what file this error is from, I get a line number and it prints the line for me! I'm bad at getting out of VIM.

Here's another error: Again not only am I getting my file and the line number, it shows me what block the error is, it is in my output that I named foobar and it's going to print out the offending line. In this case, I'm trying to add a number to string, shockingly that doesn't work. And look at that, detailed error message, this is glorious.

Last one: Unsupported block type, again, I get my file name, I get a file number, I, turns out I tried to write outptu instead of output. I get a “Did you mean,” message, like this is just beautiful. I have lost hours of my life commenting out ginormous files and then uncommenting one block at a time… yeah, you all know what I'm talking about. Life is better.

For expressions

You'all want loops. And the problem of iteration is a hard one to solve. So I'm not quite giving you blanket for loops, but we've got some for expressions that I think you'll be excited about.

We have a for expression that is for list and map transformations and for_each, which could be used in dynamic nested blocks. Oh by the way…

Dynamic nested blocks

What the heck does that mean? Here's just a basic Terraform 0.11 kind of ugly output that's taking some attributes out of an AWS instance, maybe a list of instances and kind of making it a pretty output. So let's look at that in 0.12. I have this for instance in blah, blah, blah, blah, blah. So it's taking my list, transforming it into a map, this is pretty cool.

Yay, it's a map. Good job.

That's maybe the less exciting of the things I have to show you today. If you've ever tagged an auto-scaling group in AWS, you'll know it's not like tagging anything else in AWS because you need to put each tag in individually… one at a time… tag after tag. If you're tagging resources, you don't have one or two tags, you've got a dozen. It just takes forever. If only there was a better way.

This is great. I've got this dynamic block and now I've got this locals variable up top, so it's going to let me make a for_each loop going over all of these local tags, so any dynamic nested block, we now have for expressions for it. If this doesn't make every bit of your code easier to read and reason about and refactor when necessary, I will eat a sock. I don't know where that came from.

I hope that my level of excitement over error messages and nested blocks will prove to you that I have actually spent the last several years writing Terraform code, I've been there. I've been that user doing disgusting, gnarly things to make Terraform do what I want and it's amazing to see how widely HashiCorp embraced us—the users—so that we could do these things.

Oh good, I get a water break. This is Li Shoe, she is the sweet, pretty lovely one and I just want you to remember that because there's one yet to come. She's so cute.

Act II

Let's go back to our story. To recap, our poor, exhausted engineer—me—had this bit of Terraform code with this ugly, ugly conditional statement in the prefix that was taking advantage of a hackaround that I needed to deal with weird conditional problems in Terraform.

Let's see what this is going to look like in 0.12: There are several changes here I'm going to walk through one at a time:

• The first one, I'm fully going to admit is going to be annoying, and I am sorry. count is a reserved word. You'll be excited when you see how we use it, so I've had to refactor my variable to say pet count. Sorry, slightly annoying.

• zoo_enabled, I am now really confident that my Boolean type will always be used the way I expect it to, so I've gotten weird of that weird string compare and I'm just using false as my default.

• In my resource, thanks to those first-class expressions, I was able to get rid of my quotation dollar sign curly bracket syntax, and my conditional statement is just gloriously clean and easy to read—although only as easy as I can do on a slide, sorry. As you can see, since it's Boolean I can just do a direct search, I don't need to compare a string. Terraform is now going to evaluate only the half of the conditional it needs to, so I'm okay having an empty list in my prefix_list.

I owe you an explanation from the first time I told this story. I glossed right over that. Some of you would have looked at my hackaround here and said, “Well, why didn't you just take your prefix_list and put an empty string in it and then you don't need this conditional hackaround?” Maybe a lot of you have already figured out why. In the real world, I don't have a variable that I control, I have a resource that's being conditionally created, depending on the project requirements or the developer's preferences or the phase of the moon, so I don't have control over that list. If zoo_enabled is 0, then the resources aren't created—the list is empty. That's why I had to use this hackaround, and I'm sorry to only bring this up at the end.

Look at this again. Okay, there we go, so now I can just prove to you that I can run Terraform apply again with my default file and get everything the way I wanted to. Code in slides is hard to read, so I'm going to try to really bring this to the foreground. Here is my conditional statement in the Terraform 0.11 HCL1 syntax. This is hard to read. I have written code like this so many times in so many modules and it's a pain. And this is HCL2. That's pretty sweet. I'm going to applaud because I'm excited.

What's the moral of the story here?

In Terraform 0.12, your code is easier to read and reason about. We have addressed many of the bizarre hackarounds that our users have needed. There are some things you've done, you've taken advantage of effectively accidental features, and so we've made sure these things are baked in. You'll have increased ability to write loosely coupled modules, thanks to a lot of things like those for_eachs and being able to pass an entire resource as your module output, so Terraform 0.12 is just trying to address this massive number of requests, and that's part of why it's such a big release and why it took so long. Like I said, we had to really re-factor the whole underlying foundation.

FAQ

I don't have a Q&A period scheduled, so I wanted to address what I know will be some of the highest level questions we're going to get. Okay, Terraform 0.12 alpha is available now, as you heard yesterday. I really hope you go and download it—play with it. We've bundled it with a handful of providers—those are the only providers that will work with it currently because they need to be built with the current version of Terraform.

When Terraform 0.12 is released in general, which should be later this year, it will be released with a Terraform upgrade tool, so it will go through your configuration files, make any updates necessary and print out any warnings for things you need to change manually. HCL2, we've done our best to keep it backwards-compatible so this should be a minimal upgrade, but there's going to be things you have to touch, especially if you have a lot of complicated modules.

There are a handful of features that folks have been asking for that aren't going to be in the 0.12.0 release, but they're on our roadmap. And these are all kind of module-focused, so module counts—that's why count is now a reserved word—module and resource for_eachs, and the module depends_on, so they're not in 0.12.0, but now we can build those things thanks to HCL2.

This is Cheetarah. She is a gremlin, she's not actually a cat. She is my pride and joy.

I hope that I've done a good job of expressing not just my excitement about 0.12, but the fact that we built this for you. You know, we saw how you're using Terraform, we loved it and we wanted to do everything we could to support that.

So please, I hope you download the alpha, you play around with the new language, see what kind of interesting things you can do. Give us your feedback, get involved with the community, get on GitHub and respond, and really please just go out there and have fun with 0.12.