Scaling Your Workflow With Terraform Cloud

HashiCorp Terraform engineers Robert Tillery and Mike Nomitch have been working toward the debut of Terraform Cloud for over a year—their mission has been to make adoption of Terraform for small, medium, and enterprise-sized businesses easier with a more gradual learning curve.

After many months of customer testing and research, Terraform Cloud has been developed to provide a platform of automated best practices and free features managed by our operational backend so that more teams could adopt infrastructure as code practices without having to gain as much expertise in Terraform as you would for a from-scratch DIY open source deployment.

This talk serves as one of the first in-depth demos of Terraform Cloud's free tier features. Topics covered include:

• Shared state
• Distributed execution
• Shared variables
• Slack notifications and GitHub integrations

If you're on the fence about trying Terraform Cloud, or you want to learn more about how it handles real-world workflows, this session is your first stop.

Transcript

Mike Nomitch: Hey everybody. Hope your HashiConf has gone well. We're in the home stretch. My name is Mike Nomitch. I am a software engineer on the Terraform Cloud team at HashiCorp.

Robert Tillery: I'm Robert Tillery, also an engineer on the Terraform Cloud team.

Mike Nomitch: We are excited to share what our team has been working on since the last HashiConf. With yesterday's release of Terraform Cloud, we're taking a lot of the collaboration workflows and software that we've been developing alongside our larger enterprise clients with Terraform Enterprise—and we are releasing that to teams of all sizes.

In working with these larger corporations, we found the benefits of infrastructure as code, and Terraform more broadly, don't just come when you hit apply and your infrastructure changes. A lot of benefits come in terms of streamlining your DevOps and the collaboration that happens around your infrastructure.

That's great for large organizations, and it's been a pretty easy sell for these mature organizations that have large, built out Ops teams. For smaller—let's say startup—operations, there's some overhead. Or at least some perceived overhead, in terms of how I get set up, the knowledge I need, the technology, and the cost.

Our team has been working on lowering those barriers to entry as much as possible for teams of all sizes. We want a single-person operation, a startup, an open source project—any of these teams can come onto Terraform Cloud and augment their Terraform workflows so they could get the most out of Terraform and Infrastructure as Code. That's what we're going to talk about today.

Small teams

Robert Tillery: At a previous company, before joining HashiCorp, I introduced Terraform and it was pretty smooth sailing for the first month or so. Of course, I was the only person working on it. That made it easy. I started to encounter some problems when we started to add more people to help collaborate with me.

Shared state

The first one was state storage. We had to ensure that our state was in sync across multiple people, and that presented a problem. Whenever one of us made a change, we would have to explicitly communicate to everyone else that they needed to fetch the latest version of the state.

Distributed execution

The second problem we encountered was distributed execution. It was possible for multiple people working on the same configuration to run terraform apply at the same time, and there was nothing protecting us from that. There, we also had to communicate: "I am the one running terraform apply now. Please do not run it."

Another problem that these multiple environments present is that we could be on different versions of Terraform. We had to explicitly make sure that we al upgraded at the same time.

Shared variables

A third problem we encountered was variables and secrets. We had stored our AWS provider keys on everyone's laptops in tfvars files. That is a bad thing.

Let's take a look and see how Terraform Cloud addresses each of these issues.

Terraform Cloud Free Tier demo

This is my dog, Ducky. He has founded an awesome company called DuckyCorp, and he has hired me to be the first developer at DuckyCorp.

When I come into the Terraform Cloud app, the first thing I see is my Workspaces. Here, I can see they're linked to all of these repos in GitHub. Let's say I'm ready to deploy a new HashiStack. To do this, I would hit new workspace, go to GitHub, and then select the HashiStack repo, and then hit create workspace.

Terraform Cloud is grabbing all of my configurations, and it'll let me know when it's ready. Before we can run this, we have to input some provider credentials, which Mike will do right now. But Ducky would get upset if we exposed our credentials in a recorded demo. Let's move to a Workspace where we already have all this set up.

Here we have our variables all stored in Terraform Cloud. These variables are encrypted by Vault. We've got some marked sensitive, so we don't expose the values to anyone looking at this UI. Now that all this is set up, we can hit queue plan—and Terraform Cloud is running terraform plan on our behalf. We can see the output right here.

Terraform Cloud will capture this output of this plan then feed it into the apply phase—even if the apply phase happens days later. Once this plan finishes running, we can decide to apply it or discard it. For now, we can discard it.

So far, this is nice. We've got Terraform Cloud running our Terraform on our behalf. We're not spending our CPU cycles. It's particularly nice for long-running applies. I don' have to make sure that I keep my terminal open and my laptop connected to the internet. But it starts to shine when I want to bring on another developer to work with me.

Let's say I'm ready to bring Mike on to help me with some infrastructure tasks.

Mike Nomitch: Thank you for hiring me to the DuckyCorp team. It's the single proudest day of my life, joining this esteemed organization. Let's look at this from a new hire's perspective for a second. When I get onto the team, what do I get because we're on Terraform Cloud? Without it, I could come in here and—let's say we're on AWS—I could get access to the AWS console, and I could click around, and I would have a vague idea of what's going on at this single point in time with our infrastructure.

That's a great starting point. But anyone who has worked with Git knows, one of the best features is being able to go into the past and see how things have evolved. With Terraform Cloud, I get this audit log of everything that has happened across our entire organization with our infrastructure over time.

That's great for a new hire coming on. It's very easily accessible. I don't have to dig through random log files to get it. I don't have this lost on personal machines. I have that information open to me and easily accessible. From an onboarding standpoint, that's great.

Let's click and do a run here, and I can get a little more context.

I know it has my face on it, but I'm a new hire, so let's pretend it was Robert who pushed this code. I could come in here and say, “Robert, pushed this change, which caused this apply. It was a manual trigger.” I would have access to all those Terraform plan and apply logs, and the state files, but I could also get more context by linking right into my version control system.

Robert, if you want to click the Git commit. Right here, I have access at the click of a button to the code that made this change. That’s nice to have that tightly integrated. That's great; I could see the whole history of my Terraform state files over time in Git, but because it's linked to Terraform I also have more context than I would've had otherwise.

If you had said, "Push the syntax error into master," and the terraform plan had failed for some reason, I would have that info at my fingertips. Or if you had referenced a key that didn't exist, and AWS threw an error when you tried to apply, you would have that context available for you. I'm able to augment my Git history and all our previous collaboration with more info—and it's pretty easily accessible to a new hire like me.

Let's go into the pull requests since we're in GitHub, and we could see some more benefit that this gives us. If you remember when Robert hooked up the HashiStack, he hit GitHub, hit HashiStack, and it created a Workspace. Not only did that create a Workspace, but it set up this CI pipeline for us automatically.

On any merge and any pull request, I automatically get what we call speculative plans that will run on all of these commits. I don't have to think about how this will change my infrastructure. I don't have to be running terraform plan manually on my machine. If someone forgets to do that, any member of the team can jump in here and see what the change looks like—not necessarily like an apply in master, but on any pull request.

That's helpful, again, for gaining more context as a new user. That's great; I've poked around everywhere, and now I have an idea of what the infrastructure looks like—what this new company looks like. I'm ready to make my first change and help DuckyCorp. Robert, what should I be doing, here?

Robert Tillery: Ducky would you to figure out a way for you to improve the overall performance of our application.

Mike Nomitch: That's a good task. I think I know what that means. I'm pretty sure he wants me to bump the instance size from a T2 micro to a T3 double XL. That will make things run faster. I'm confident that's what he wants; who needs code review? Let's merge this.

Robert Tillery: Fortunately, I've been notified.

Once it starts running, we'll get a notification in Slack that it's running. Fortunately, I've been notified that Mike has a change that he would like to apply to our production environment. Let's go check it out.

That's not what I had in mind. I was hoping you'd come up with a different solution to that. Let's discard this run, and I'll leave a note to Mike as to why.

Now I think, "All right, it might be a good time to get Mike working on the staging environment."

To do that, we can create a new Workspace. We will click HashiStack again, and name it something different. This Workspace we will link to a staging VCS branch. For some teams, this branch-based workflow makes a lot of sense and is very natural.

Other teams may have multiple Terraform environments spread across the repo and in different directories. Luckily, Terraform Cloud supports that, too. If we had environments set up that way, we could specify the Terraform working directory and make sure that we have the run-triggering set up properly. Then only changes that occur in that directory will trigger runs.

Another good feature of Terraform Cloud is locking. Say I am investigating a problem in our Lambda stack, and I want to make sure that it doesn't change. I can lock it by going to the locking page and then hit the lock button. Let's see what happens when Mike tries to make a change to the Lambda stack now.

Mike Nomitch: From my perspective, I'm changing this code locally. Let's say I'm in some café; I've turned Slack off. Robert and I haven't communicated, and I don't know that he's inspecting something and doesn't want anything to change.

I hit apply, and I push my local changes up. It’s worth noting that Terraform Cloud can enforce that all of your changes go through VCS. You can set this up so that this would immediately fail and say, "GitHub master is the only place the changes should occur that affect your infrastructure.” Or you can force a CLI-based approach like this.

So even though I'm working on my command line interface like I'm used to with Terraform, it's using the remote plans it applies. Why would I want to do that? This is one reason. It says waiting for the manually locked Workspace to be unlocked. Without Robert having to explicitly communicate to everybody on his team that this infrastructure shouldn't change, it automatically stops me from making this change.

I can say, "Hey, Robert, are you done with that?"

Robert Tillery: Go for it.

Mike Nomitch: He'll come over here, hit unlock, and now we can see that my change will go through. One of the benefits of using Terraform Cloud is you get this workflow that you're potentially used to—and you make sure that you don't have two applies that happen at the same time and leave your Terraform state files in a corrupted state. It's nice to have when collaborating with a team.

That's it for our demo.

Terraform Free Tier recap

Let's step back for a second and think about what we got from being on the Terraform Cloud Free Tier with two members. The first thing that happened was the Terraform state—the issue around syncing that; I didn't have to set up a Consul backend, or an S3 backend. That was a solved problem for me.

Our remote runs; they executed off my machine. I didn't have to worry about wasting CPU cycles on that—and we could make sure that we didn't have conflicting runs that left our infrastructure in a bad state. Furthermore—and I think this was important: you get this history of everything that's happened to your infrastructure in an easily accessible place.

Then there are all these nice quality of life improvements, like shared variables, VCS integration—you just push it up to GitHub, and everything's running. Then also events—pushing to Slack, or general webhooks to keep all your team in sync.

That's great. With two developers, we're running swimmingly, but DuckyCorp is going to scale huge, and we're about to hire 100 new developers.

Terraform Cloud for larger teams

Robert Tillery: With larger teams, DuckyCorp has encountered a whole new set of challenges, and we'd like to see if Terraform Cloud can help us. The first thing is we have lots of developers now, and we have multiple different projects going on, and lots of Workspaces in Terraform Cloud.

Team Management

We would like to group users and give them particular team access to very specific Workspaces, and you could do that with Terraform Cloud.

You could give read, or plan only, or apply, or administrator access to each Workspace. You also get a user dedicated for your CI/CD pipeline.

Shared configuration

Another thing that Terraform Cloud can help us with, with a bigger team, is shared configuration, which is large building blocks of infrastructure as modules. Terraform Cloud provides the Private Module Registry to store this shared configuration that we can then use across all of our Workspaces.

Policy as Code

Then a third thing is Policy as Code. Sentinel allows us to apply policies to our Terraform configuration as either mandatory or advisory overridable. For instance, let's say DuckyCorp has a bunch of different projects going on, and we have one well-known way to build a VPC. We can wrap that up into a module and then use it in every single project that we work on. It's very simple to use a module. You follow the provisioning instructions here and provide the inputs.

This is an example of a Sentinel policy that could be used to prevent Mike's attempt to solve our performance problems by bumping the instance type. The way this works is you import the TF plan. You define the allowed instance types, and then we iterate through all the instances and make sure that the instances in our Terraform config are in the allowed instance types.

If Mike had tried to apply his change, and the Sentinel policy were in effect, he would be blocked after the plan phase—before the apply phase. Sentinel also provides some other good options for policies. For instance, you could require that all of your instances have a particular tag associated with them.

You could also apply policies to your security groups to make sure that your firewall isn't open to the outside world. Then also my favorite is time-based—where you require no infrastructure changes on Fridays.

The stages of Terraform adoption

Mike Nomitch: As we scaled from Robert to me and Robert, to potentially many, many users, we got different benefits at each of these stages from Terraform Cloud. The first benefit that you see when it was just Robert is a quality of life, ease of use improvement, on top of Terraform. You get to push up your VCS, and it keeps your infrastructure in sync.

You want that to be one-to-one as much as possible—and you want to reduce the friction to getting there. Getting that integrated into your CI as quickly as possible, we think, is pretty important—and it makes your life easier and better.

The second stage was, "Now we're collaborating with someone." There are a whole host of little frictions that come into play when that happens—whether that's sharing state, having two applies happen at the same time, sharing environment variables.

If one of those environment variables got leaked for some reason—and we had to cycle those—it's nice to have that in one shared space. Then I think, most importantly out of all of this—you get that shared context, and you have something that augments your normal collaboration flow via GitHub or whatever VCS provider you're using.

We have integrations with Bitbucket and GitLab as well—and you get that extra context for all of your team when you're onboarding them. That was great as our team scaled, and then as we got to a huge stage—as Robert mentioned—there are all of these things that don't scale as you get to an even bigger organization. You don't want 20 different ways of configuring some auto-scaling group, or VPC, or cluster. You want that the be automated. You also don't want to be manually checking to make sure that you have the right security rules in place, or best practices. We love automation here, and Sentinel allows you to do that in a more automated fashion.

We're excited to throw this out there and see how smaller teams can use this—and teams of any size can use this. I'm excited—and I think our team's excited—that a lot of what we showed is free. All the stuff that we showed in the demo, in the live demo portion, is available for free.

We like that because the open-source community has given so much to us. If you look at all the providers that have been written, it's incredible. We don't want to throw this open-source tool out into the world and say, "Good luck using it once you have three or more people." We're glad that we could take a lot of learning from organizations that have been using this, and provide a pathway for new organizations to get ramped up—and for everybody to start using that as their org grows.

I highly encourage you to check it out. Terraform.io. Go there. Play around with it. Again, a lot of this is free, and thanks a lot.

Robert Tillery: Thanks.