Deploying Terraform Enterprise in Air Gapped Environments

We assume that organizations utilizing or deploying workloads into the cloud have unfettered access to the internet. This, of course, isn’t true. A great number of systems exist in what are called “air gapped environments,” and the industries that utilize them span public sector (government and military), finance, energy, and more. If you’re reading this, that probably means you’re familiar with these types of environments, but for those that aren’t, the bottom line is this: Air gapped environments are networks that are isolated from other networks, usually both physically and logically. That means no internet. No outside connectivity.

Air gapped environments therefore present some unique challenges to the installation and maintenance of applications, especially when those applications are typically installed using modern methods that depend on internet connectivity.

Those auto-updates that fetch new versions from your vendor’s website… yeah, those aren’t going to work. However, all is not lost.

HashiCorp Terraform Enterprise is a provisioning platform that accounts for air gap restrictions. In this blog, we’ll review some of the mechanisms it uses to do this:

• Install Types: Online vs Air Gapped
• Performing an Air Gap install
• Creating Terraform Enterprise update bundles
• Deploying Terraform Enterprise update bundles

Install Types: Online vs Air Gap

Terraform Enterprise installs using either an online or air gapped method and as the names infer, one requires internet connectivity, the other does not. Online is easier and requires less preparation, but both result in fully functioning Terraform Enterprise environments. Online will evaluate the type of infrastructure you are installing onto, fetch and install prerequisite software (e.g., Docker), and then fetch and install Terraform Enterprise. Air gap requires some additional effort.

The Terraform Enterprise installation script file must first be downloaded and made available to the target system. This file is usually stored/hosted within the airgapped network for future use and copied to the specific host. As this script can’t access the Docker website to install Docker, it is necessary to manually perform this function. Most organizations will maintain copies of the Docker installer within the air gapped network and automate this step or have Docker baked into a base image.

Finally, the install script will need access to the Terraform Enterprise installation files, and as these also aren’t accessible via the internet, it is necessary to stage them someplace accessible within the air gap network. To allow for this, HashiCorp produces ‘air gap bundles’ which are individual files, per Terraform Enterprise release, that contain all the necessary bits and pieces that would otherwise be fetched over the internet.

Performing an Air Gap Install

The following is a high-level view of what the installation steps/screens would look like for an Air gap install. This isn’t a replacement for the detailed installation documentation; be certain to review the documented installation steps and ensure you review all information specific to your platform before performing an installation.

After ensuring all prerequisites are met, execute ./install.sh air gap command to begin the install. Upon completion of this step, you will be presented with a URL to continue the install via a web-based GUI.

From the web-based GUI, upload your SSL certificate, or use a self-signed cert.

Upload your license file, supplied to you via your HashiCorp sales team.

Select "Airgapped" installation type.

Specify the air gap bundle file you will have downloaded from the HashiCorp site using the URL supplied to you by your HashiCorp sales team.

Identify any missed prerequisites.

Monitor installation until complete.

Upon completion, click the “Open” link (beneath Started / Stop Now) to access Terraform Enterprise and create your admin user.

Once installed, the process for setting up your admin user and interacting with Terraform Enterprise is nearly identical to working within an online environment. There is one key difference that you’ll want to incorporate into your internal workflow. And that pertains to how you access and utilize Terraform providers and new versions of the Terraform binary. A Terraform Enterprise install does not include any providers. In an online installation, when Terraform Enterprise executes Terraform code that specifies specific versions of the Terraform binary or provider files that aren’t installed, Terraform Enterprise will reach out to the HashiCorp servers and fetch them; this of course won’t work in an air gapped installation. And this is why HashiCorp provides a mechanism to create Terraform bundles.

In simple terms, a Terraform bundle is a zip file that contains at least a specific version of a Terraform binary, and optionally provider files. This bundle must be created from a system that has internet access so the requested files can be fetched and bundled together. This bundle file can then be transferred, often via sneakernet into the air gapped environment, and then installed onto the Terraform Enterprise server.

Creating Terraform Bundles

A Terraform bundle file, in its most simple form, is simply a zip file, with a specific version of the Terraform binary at the root level, and the corresponding provider files also at the root level. These files can be downloaded directly from releases.hashicorp.com. You’ll want to name the zip file following a standard that references the Terraform binary version, platform, and architecture.

This method works when you have an infrequent need for small bundle files that include few providers. If you need to generate bundles that contain many providers and you wish to generate these frequently, — both common scenarios especially within air gapped environments — HashiCorp maintains a tool to simplify the process, and more importantly, enable you to maintain a record of what you have placed within the bundle. This is appropriately named the Terraform bundle tool, as referenced within our docs.

This tool allows the user to specify, within a configuration file, what Terraform binary version and what providers to fetch. This configuration file can be maintained in your version control system to ensure you have a record of what bundles you’ve built and when. In the image below, I’m executing this tool from a published Docker container, saving myself the time of compiling the tool. The result is a zip that I’ll make accessible to Terraform Enterprise, usually via a web server within the air gapped network.

Deploying Terraform Bundles

The Terraform Enterprise web interface includes a screen accessible from Site Admin > Terraform Versions that displays the different versions of the Terraform binary installed.

It is important to note that the entries on this screen are only references to 'potentially' available versions of the binary. TFE doesn't include any binary versions during the installation and as such, it is up to you to decide which version(s) you will make available. In an environment with internet access, these could simply be fetched as needed. In an airgapped environment, however, you'll want to generate bundles for those versions you wish to enable and host those files, perhaps on an webserver within the airgapped environment.

On this screen, if an admin user clicks the "Add terraform version" button they will be presented with a screen to supply the relevant information in order to reference the Terraform Bundle file previously generated. Once referenced, the admin can decide to make this bundle available to all users to be consumed, or limit its use, perhaps for initial testing. The URL field should reflect where TFE will fetch the bundle from and the checksum should indicate the correct value for the bundle.

As is typical of all things Terraform, everything is accessible via API. As such, all of the functionality described thus far, plus additional commands such as deleting bundles from the Terraform Enterprise server can be performed via API. Our API documentation provides details and examples that you could leverage to build an automated workflow to periodically produce a bundle from the latest binary releases. This must, of course, take place on a system with internet access.

To further simplify this process, the Provider Network Mirror Protocol was released for use with Terraform CLI v0.13.2 and later. This allows you to establish, as example, an artifact store within your airgapped environment that you maintain provider files on. Specifying the URL for this server within the Terraform CLI configuration allows providers to be retrieved as necessary from this server.

Summary & Screencast

Working within an air gapped environment shouldn’t be viewed as an impediment to realizing the benefits of infrastructure as code. Understanding the differences when deploying and maintaining your Terraform Enterprise servers in these environments will greatly benefit your effort and are key to achieving success.

To see this process demoed in a webinar, watch the video embedded below:

You can also watch or read the transcript for this case study where Terraform, Vault, and Consul are used in an air gapped, government IT environment:

How Many Tools Does It Take to Build a Cluster: Terraform, Vault, and Consul in Government IT