Managing Terraform Cloud With PowerShell

Learn how City National Bank simplifies repeatable tasks on Terraform workspaces, variables, and modules at scale using a PowerShell API wrapper.

Related GitHub repo for this talk.

»Transcript

Hi, welcome to Managing Terraform Enterprise or Cloud with PowerShell. My name is Paul Kelly. I work at City National Bank. I bring about 20 years of experience in technology, and 10 of those years, I’ve been working a lot with PowerShell. 

»Overview 

I wanted to give an overview of what we’re going to talk about today. First, I’m going to go through PowerShell cmdlet syntax if you’re not very familiar with it. But the bulk of the presentation’s going to be around managing your Terraform organization at scale. Whether that’s individual workspaces, or modules, or many of them at once. 

»Powershell CMDLETS

But let’s jump in. PowerShell cmdlets, always start with Verb-Noun. You can say New-TFWorkspace, Get-TFWorkspace, Copy-TFWorkspace, whatever action noun that you want to do with that workspace or module. You can also string together commands with pipes. 

You could do a Get-TFModule GKE-Latest, and you can Remove-TFModule. If you want information on each of the commands, they come complete with Help. There are examples. You can look at the different parameter types for each of the commands. 

As you’re making live changes to your Terraform environment, you want to also make sure that you confirm. If you’re going to delete all your workspaces, it’ll give a confirmation to make sure that you don’t wipe out your environment. So, if you’re proficient, then you can use aliases to execute these commands at a quicker rate. 

»Why Do You Want To Manage at Scale? 

The business need we have is to quickly make sense of your Terraform organization. You need to manage many workspaces. There are all kinds of settings, many variables to manage.

You also want to make sure your private module registry is clean and in a state you desire. 

The solution is you get instant feedback using PowerShell connecting to the REST API, and you can leverage PowerShell’s robust pipeline to manage hundreds of objects customized to your specific needs. You can even schedule those jobs for full automation, so you don’t have to worry about doing repetitive tasks on a weekly or monthly basis. 

»Demo #1: Managing Workspaces at Scale

With that, let’s jump into a demo. I’m going to run through backing up your workspaces. A common task is upgrading your Terraform version. Again, you can do that with one workspace or many workspaces at a time, and you can also easily compare your workspaces to make sure that they’re standardized across your organization. 

Open up a PowerShell window. You can use Windows or Mac. You can run, get command module Terraform, and you’ll see all the commands related to this Terraform module you just installed. You can do get alias where the source equals Terraform, and you’ll get all the correspondent aliases for those cmdlets.

You can execute a get command *Terraform workspace* and get all the verbs related to that workspace. You can copy, get, lock, create a new Terraform workspace, remove, set, and unlock.

»Getting Started 

First off, you’ve got to get connected. Hopefully, you’re already using Terracreds, you can do a get Terraform workspace, connect to Terraform first. Then you get Terraform workspace, and it’s going to return all of the workspaces in your organization at once. As you see here, you have Name, Version, Working Directory, and Branches. 

You can measure how many workspaces. Doing this to the GUI, you can already see the benefit of doing it at the command line. Another view, which is just native to PowerShell, you can outgrid view, and then you can easily sort and filter these. If you want to copy and paste this into Excel, it’ll retain the tab spacing and gives you some good insight — another way to view your workspaces.

»Filtering

You can also do filtering, so get Terraform where the name is like *-Tony*, and it’ll get all of Tony’s workspaces. You can see here very quickly the power of filtering. Depending on your naming convention, you can also use any aspect of this. You can use the Name. You can use the Version, Working Directory. 

Here, another example is where the name is like Prod. Then you could quickly get an understanding of all of your Prod workspaces. I just threw in an alias. GTFW is an alias for get terraform workspace. You can measure all of your production workspaces with that command.

»Grouping 

You can also group, so you can get Terraform workspace, and you can group it by version. Here is great because now you can see exactly what version all of your workspaces are running on and how many of each. 

If you need to upgrade from version 0.11, or you want to get onto version 1, you can group them and get good visibility into your environment. You can group on any of the various fields. Here’s an example of branch. You can make sure that, depending on your branching strategy — if it’s Dev or Prod or just Master — you can make sure that the respective branches are used.

With a simple CLI export, you can export XML into a backup file. I’m going to import that and display the first five lines, to show that the backup file was exported. I just saved it to a backup variable. You can select the first five, and you’ll see, now you have an XML backup. 

We’re going to look at the first one. First, the table is just going to display what’s relevant, but you can do a format list and view all of the different associated properties. 

In the same way that you can backup to XML, you can also export this to CSV — and this is a little more friendly using Excel versus XML. XML, you probably want to use it for backing up the objects. It’s easy to import those objects, but then with Excel or with CSV, this is more of a good reference point if you just want to parse through and look at the data. 

Or, you could do an export. You can attach it to an email and send it out. You can dig into the specific perimeters that are within your workspace. Little SNAFU with Excel, but we’ll just skip over that.

»Upgrading Your Terraform Versions at Scale 

We’re going to get workspace, pipe that to group Terraform version. Again, we just see what versions we have on the box. Maybe I want to grab just version 0.13, and we’ll see here that we have four workspaces that are using version 0.13. 

You could upgrade them all at once, but we want to be a little more prescriptive in this. We want to get one of them. Let’s choose a Dev workspace. Let’s try and upgrade Joanne first. We’ll just get that one specific workspace. Then we can set that workspace version. 

We’re going to set that to version 1.08, and there you just saw it prompted for confirmation. We want to always confirm we’re not going to just do things blindly, but we have that confirmation prompt built into PowerShell. Then from the command line, we can start a run on that. We’ll let that run. We’ll just assume that it all went well. 

Next, we’re going to get whatever remains. Instead now of four, you see three, and we will now set it. We’ll hide that confirmation. Now all of them are on version 1, and there are no longer any workspaces on version 0.13.

And we’ll do a comparison with what we backed up. Notice our backup. We have that as a reference point, but now into our form, we don’t have any version 0.13 We’ll just confirm with a group. We’ll get Terraform workspace. We’ll group that by version — notice they’re no longer 0.13. 

»Standardized Branching

Another thing we can change — another example — would be maybe we want to standardize on some branching. We can look for all of John’s workspaces. We’ll notice that he’s not following the branching convention. We want to set those other branches to use Main so that they’re standardized — so we’ll confirm all of them. When we run and compare it again, we’ll see that all the branches are now building off of the main branch. That concludes our demo. Thank you for watching.

»Demo #2: Managing Individual Workspaces

In addition to managing workspaces at scale, we can also manage them individually. In our next demo, we’re going to dive into creating a new workspace from a single command. We’re going to clone a workspace, including the variables, and manage those workspace variables. If you need to create new ones, set existing ones, or remove no longer needed variables. 

Also, with that specific workspace, we’ll be able to back up the state as well as view it. This next demo I’ll do on my Mac. We’re going to look at working with individual workspaces. I pulled up the GUI so we can follow along. I search for any workspace of Fahim’s, and we see that there are no workspaces there. 

»Create a New Workspace

With the one-line command, I’m going to create a new workspace. I confirm it. I can set all the parameters, and instantly you see that new workspace created in the UI. Let’s run-through and critical workspace variables. I’ll go to the variables, and we’ll see that there are currently no variables as this is a new workspace. 

I want to set a couple of variables. You simply specify the workspace you want to modify and use the Terraform workspace cmdlet, and you set the key as well as the value. You can use the HCL switch. You can use the sensitive switch and hide the values. We’re going to create a couple of dummy workspace variables for Fahim here. One through. This one takes a refresh, and you can see now we have these three workspace variables provisioned into this workspace. 

»Cloning a Workspace

Now the fun part, what if you want to clone this workspace? Now you can use the copy Terraform workspace command, and you just simply specify the source and the destination. On the left side, you see that is are only one Dev environment. Now we’re going to do a clone, and we’re going to name this one Prod. Now we have a Prod workspace. All the workspace settings are identical, as well as the variables. 

We want to go in and look at what these variables are so we can update them. Notice that the environment variable is still Dev. We want to just switch that over to Prod. You can click through in the GUI, or again, just run a simple PowerShell command. That’s updated to Prod. 

Say you’ve done testing, or you’re decommissioning, you can just simply use the remove. Of course, doing a confirmation to make sure you don’t accidentally remove something you don’t want to. We chose yes for one and no for the other. It’s going to go through each of the objects and handle them accordingly. 

»Demo #3: Workspace Variables

You can get, create new, remove or set. Doing another little quick demo. We’re going to get all the workspace variables for Freddy now. This is a great way to compare between two workspaces. Say you have multiple environments, but there are some inconsistencies. You want to make sure that you set those variables the way that you want them to be. Notice that there is a typo there. There are two Lambda functions, so let’s remove one of them and let’s go over. We’ll just start up the UI to compare and see it live.

Now we’re going to create a new variable, and we’re going to set that brand new variable to value 1. Say you do your testing, and now you want to set that. You just simply use the set verb instead of the new verb for Terraform workspace variables.

And that’s it, simple workspace management. Now it can compare the two, and looks like I had a little error on my side. I had a typo for my typo, so let’s do a copy and paste and fix that — and then do a compare for real and make sure that it’s accurate. Now the two environments are identical, including the HCL property or the sensitive.

»Demo #4: Get The Terraform State for a Specific Workspace

As you do different plan applies, you’re going to get multiple Terraform states saved. You can look at all of them in a list or table format. 

Here you can see the different versions of the state saved here. For simplicity, I’m going to save the current. You can use the current switch, and that’ll grab the latest — and I’m going to save that to the state variable. Here’s an output of that. As you can see the different properties, there’s windows created, the size of it, what modules are used, what resources are in there, and you can parse through those properties and find these are all the modules. 

Then you can also do a table view of the various resources. Again, you can quickly see what resources are there, and you can simply start process on that download URL, and that’ll download that state file. Again this is something you can just set up a cron job and — on a regular basis — backup state files if you desire that. I just did a get content to look through and view that state file that we just downloaded.

»Demo #5: Run Management

Run management is critical. In the next demo, we’re going to look at all workspaces. You can pull out the run status of the entire organization. That gives you the ability to discard failed runs. You can also plan and apply them. You can lock workspaces. You can unlock them. You can manage all of that with very few commands. 

Jumping into run management, I wanted to show a little bit more about the comparison — here’s Linux, I’m on my Mac. With PowerShell, you can do an LS, and it’ll give you output that you’re used to. We can run PowerShell, and I’m going to install the module, and connect to Terraform.

I’m going to use the -Terracred switch, which will let me authenticate via Terracreds. There’s a link at the end of the video where you can go to download that.

Again, the difference between LS and DIR PowerShell, they programmed it in a way that will output whatever you’re used to seeing — so if you come from a Linux or Mac environment. We’re going to get the various versions or the different verbs for Terraform run. 

You can get, set, and start. We’re going to get all the current states. We’re going to group them, filter them, and even discard some of the failed plans. With this — since it’s going to parse through each workspace, and it’s going to look at the run state of each — I like to usually save this to a variable, so you’re not hitting the server each time. Get Terraform workspace for each of them, get Terraform run where it’s current. We’ll let that run for a moment.

Then we can execute that new variable you just created and see the output of that. Then, we can go through and manipulate it. You can group it or look at the various statuses and filter it. 

It finished. Here are the different plans that were run, and now we can group them. We can group them by status, and you get a good snapshot of all of the run states in your entire organization. 

You can sort it by the status not equals applied, or status equals planned, or if it’s like something, you can use wild cards. You can even add in some other PowerShell cmdlets to filter and say “I want to get the created date if it’s older than three days.” Maybe you might have a stale plan running that you might want to get rid of. We’re going to pull up the GUI to be able to compare, and you can see you have a Prod and a Staging environment. Both are in a planned state.

You can go through and look at what actions you can set. You can look at the help file, and you can see that you can apply discard, cancel, and sometimes you might need to force an execution or cancel.

»Set Terraform Run Action to Discard

That’s where it’s planned. Let’s set the terraform run action to discard. Because of this, you want to make sure the confirmation is added so you don’t break anything. I’d ask you if you want to confirm discarding this particular run, and you can hit yes, and we’ll discard both of those. Now you have your Terraform environment — it doesn’t have any runs. 

Now we’re going to do a get run on a specific workspace. When you execute, get terraform run — and here the example we’re using is as if — you’ll see all the different plans that have been planned for that workspace. You can use the current switch, and that will show you the latest run — which in this case was discarded. You can see the message. It was cued by PowerShell.

We’ll do a comparison with the GUI, and you can see in there the run status. Currently the workspace is unlocked. But say you want to lock down the workspace to prevent it from having any plan or apply — say maybe your production workloads — you want to help secure those workspaces and lock them; you can do a simple lock terraform workspace. 

As you can see, now that it’s locked, we can run a plan, but nothing’s going to happen because it’s not allowed. We’ll give it a moment. You can see the plan is pending, but it’s not able to execute. We’ll have to go, and we can look at the current, and we’ll see that it’s pending and it queued but not unable to execute. 

We want to unlock it. We’ll hide the confirmation, and you can do that again, just with a simple command. You’ll see that the lock will be removed, and the plan will automatically start running. That lock there is because it’s in the middle of a plan. That’s Terraform locking it. We’ll give that a little bit of time to run, and then once it finishes — as it just did — we can do a set apply, and this will apply the plan

You can go through — and that message that we said — you can see that it has that message with the plan. You can see if the different variables or the different resources that Terraform is provisioning. Shout out to Asif for setting up this workspace for me. Thanks. 

Now that plan is running, and it’s applying. We’ll give that some time. As that runs, I’ll pull up AWS. As you can see, there are no running instances. Then we’ll be able to go into AWS and see Terraform provisioned all of those resources through code. 

While that runs, let’s look at a better use case that would be helpful. You know, instead of just managing one workspace, we can again manage it in bulk. Let’s get the first five of these workspaces — or maybe get one of the workspaces — and list all of the parameters that are saved to that workspace. You can look at the locked parameter and see if it’s locked or not. But you can look through and all of the various settings you can filter on any of these.

In this case, we’ll go to lock. We’re going to get all of the locks, all of the workspaces in the organization where locked equals true. You can see there are currently no workspaces that are locked. 

Let’s go and look for our Prod workspaces. You have a handful of Prod workspaces here to play around with, and if the name is like Prod, I want to lock them, and it’ll go line by line on each object. You can say yes to the first one, say yes to the second one. If you have hundreds of workspaces, then you can just press A and apply it to all of those workspaces. You can see that the locks popped up. We’ll hit all and apply it to all workspaces.

Now we’ll just confirm also through the UI, or we’ve seen in the UI. Now we want to confirm it also through the command line. And you can do a get where locked equals true, and then you can unlock them — one or all of them. 

In this case, we’re unlocking Oscar’s Prod workspace, and we can get Terraform where equals locked and unlock them all and suppress the confirmation to make our lives a little easier — one less click. Let’s come back to look and see if our plan is complete, and we can see everything’s been provisioned within Terraform. 

Now we’re going to go over to AWS, to our refresh and we can see that those two VMs were provisioned. That concludes our video. Hope you enjoyed watching, have a good one.

»Registry Module Management 

There is still lots more content to cover, but for the sake of time, I just had to leave this one as a slide. There’s no demo for the module registry, although you’ve seen the other commands, and from that knowledge, you’ll be able to easily see how to manage the module registry. You can use Get-TFModule, and you can list the entire registry. 

You can look at a specific version. You can look at the latest version of a specific module, and you also have the ability to publish and remove modules. This is key as you are developing modules to be able to test. Another option is to remove modules. Maybe five versions back you want to just remove deprecated modules. You can schedule a job to automatically take care of that registry hygiene.

»Conclusion, Links, and Thank Yous 

Now that you’ve seen the power of PowerShell, pun intended, you can follow this. To get started, just install the PowerShell module for Terraform. It’s the specific Terraform, not PS Terraform, not these other modules that are out there. 

A big shoutout to Tony for creating Terracreds. You’ll want to use that to manage your credentials. It supports cross platforms, whether you’re on Linux, Windows, or Mac. It’ll use that local OS credential store to save your API token. 

Lastly, you want to go into Terraform and create that token that’ll give you access to the API, and you’re all set. All you need to do is just follow these quick, easy steps, and you’ll be off to the races.

I want to give a special thanks to my wife for holding it down as I work — and my three beautiful daughters. I put up a famous scripture here from our marriage, it’s, ”Be completely humble and gentle, be patient bearing with one another in love.” And she definitely bears with me — and thank you for bearing with me.

I hope you enjoyed this presentation and it’s helpful for you and your organization. I also want to give a special shoutout to the City National Bank team. When I joined the bank, I had no Terraform experience, and it was Fahim and Freddy who showed me the ropes there — Terraform Jedis. They do a great job along with Asif and Tony. Asif, special shoutout there as well. He set up the AWS infrastructure that we provisioned here today. Then we have Tony, who created Terracreds. I also want to make note of some management folks, as well as the cloud team. Thanks for all you guys do. Super grateful to be working with such a great team. 

I can’t leave out the HashiCorp group. Will Anderson, thank you for all your help. It’s been great talking to you about PowerShell and your Microsoft days. It’s been a blast creating this and thanks again for your help and the rest of the team there. Hope you enjoyed the presentation. Hit me up on LinkedIn, or at GitHub. Have a good one.