Version-Controlled Infrastructure with GitHub & Terraform

For a more up to date tutorial on using Terraform and GitHub together, read A Practitioner’s Guide to Using HashiCorp Terraform Cloud with GitHub.

At HashiCorp, we build open source tools that enable organizations to provision, secure, and run any infrastructure for any application. One of those tools is Terraform. Terraform enables you to safely and predictably write, plan, and provision infrastructure as code. It codifies APIs into declarative configuration files that can be shared amongst team members, treated as code, edited, reviewed, and versioned.

»Terraform Workflow

Here is an example Terraform configuration for provisioning an instance on Digital Ocean and assigning that instance's IP address to a DNS record using CloudFlare.

resource "digitalocean_droplet" "web" {
  name   = "tf-web"
  size   = "512mb"
  image  = "ubuntu-16-04-x64"
  region = "sfo1"
}

resource "cloudflare_record" "web" {
  domain = "hashicorp.rocks"
  name   = "demo"
  value  = "${digitalocean_droplet.web.ipv4_address}"
  type   = "A"
}

Terraform automatically handles the order of operations and parallelizes operations where possible. This parallelization enables provisioning of incredibly large infrastructures in minutes or even seconds.

Terraform can also execute a plan (dry-run) to plan and visualize the impact of changes before executing them. This is more beneficial when modifying existing infrastructure so you can easily understand rollout impact.

$ terraform plan
+ cloudflare_record.web
    domain:   "hashicorp.rocks"
    hostname: "<computed>"
    name:     "demo"
    proxied:  "false"
    ttl:      "<computed>"
    type:     "A"
    value:    "${digitalocean_droplet.web.ipv4_address}"
    zone_id:  "<computed>"

+ digitalocean_droplet.web
    disk:                 "<computed>"
    image:                "ubuntu-16-04-x64"
    ipv4_address:         "<computed>"
    ipv4_address_private: "<computed>"
    ipv6_address:         "<computed>"
    ipv6_address_private: "<computed>"
    locked:               "<computed>"
    name:                 "tf-web"
    region:               "sfo1"
    resize_disk:          "true"
    size:                 "512mb"
    status:               "<computed>"
    vcpus:                "<computed>"

Plan: 2 to add, 0 to change, 0 to destroy.

Once satisfied, we can apply these changes, creating real production infrastructure.

$ terraform apply
digitalocean_droplet.web: Creating...
  disk:                 "" => "<computed>"
  image:                "" => "ubuntu-16-04-x64"
  ipv4_address:         "" => "<computed>"
  ipv4_address_private: "" => "<computed>"
  ipv6_address:         "" => "<computed>"
  ipv6_address_private: "" => "<computed>"
  locked:               "" => "<computed>"
  name:                 "" => "tf-web"
  region:               "" => "sfo1"
  resize_disk:          "" => "true"
  size:                 "" => "512mb"
  status:               "" => "<computed>"
  vcpus:                "" => "<computed>"
digitalocean_droplet.web: Still creating... (10s elapsed)
digitalocean_droplet.web: Still creating... (20s elapsed)
digitalocean_droplet.web: Still creating... (30s elapsed)
digitalocean_droplet.web: Creation complete
cloudflare_record.web: Creating...
  domain:   "" => "hashicorp.rocks"
  hostname: "" => "<computed>"
  name:     "" => "demo"
  proxied:  "" => "false"
  ttl:      "" => "<computed>"
  type:     "" => "A"
  value:    "" => "192.241.218.201"
  zone_id:  "" => "<computed>"
cloudflare_record.web: Creation complete

Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

»Terraform Pro for Collaboration

The Terraform CLI works great for individuals, hobby projects, and small teams, but like most CLI tools, it tends to not scale well without a centralized coordination system. The objective is to make safe, predictable, and transparent infrastructure changes across a team regardless of the number of collaborators.

This challenge is not unique to Terraform. In fact, Git suffers from the same problem. Git is an excellent tool for managing source control, but lacks support for teams, permissions, ACLs, and reporting. Just like GitHub fills those requirements for Git, Terraform Pro bridges that gap for Terraform.

Terraform Pro is to Terraform what GitHub is to Git

Terraform Pro is designed for teams to collaborate on and organize many Terraform states, configurations, modules, and variables. At the core of that collaboration is our integration with version control systems like GitHub to take infrastructure as code configurations and turn it into real infrastructure on any provider.

»Terraform Workflow with GitHub

After we have connected Terraform Pro to GitHub using the standard OAuth workflow, GitHub can automatically notify Terraform Pro of changes to code at the Version Control Software (VCS) layer. These change notifications, in the form of webhooks, automatically trigger a plan phase. Terraform Pro controls the version of Terraform, the ingress and egress permissions, and securely stores and manages provider credentials.

Here is the same code from the previous example, but it is now committed to a GitHub repository that is connected to Terraform Pro.

We can link this GitHub code repository to Atlas by authenticating with GitHub using OAuth, and providing Atlas with the information to access this repository.

After linking, when new branches are created on this repository, Terraform Pro will automatically execute a dry-run on the configuration changes and report back the results to GitHub through a Pull Request. Here is an example GitHub Pull Request that shows Terraform Pro is currently executing the plan phase of our changes.

After the plan is complete, Terraform Pro will report the results back to GitHub. Assuming the changes were valid and would successfully apply, a green checkmark will be visible.

Clicking on the details link takes us to Terraform Pro where we can see the exact plan output, just as if we had run Terraform ourselves locally in the terminal. Other members of our organization have visibility into this plan, and they can provide their feedback in Terraform Pro or GitHub.

Assuming these changes are approved, we can merge the code on GitHub. This will automatically trigger a new plan on Terraform Pro, and notify the appropriate parties via Slack, email, or a number of other notification methods. Once a plan has reached the default (master) branch, it can be confirmed, which will execute the apply phase of Terraform. This will change real infrastructure.

When we click "Confirm & Apply", Terraform Pro will execute and modify real infrastructure resources on the cloud.

As expected, if the Terraform plan fails, perhaps due to a syntax error or invalid permissions, a red X will be visible, and GitHub will warn us about merging these changes.

»Consistent Workflow

One of the many benefits of treating our infrastructure as code is that we can leverage years of tooling, workflows, and technologies used by application developers. Because our entire infrastructure is contained in plain-text files, we can easily leverage tools like Git and GitHub to version control our Terraform configurations. Combined with Terraform Pro, we are able to apply continuous integration on our Terraform configurations.

The first principle of the Tao of HashiCorp is Workflows, not Technologies. While Terraform itself is a technology, its goal is to provide a mechanism for safely and predictably changing infrastructure in a way that is technology agnostic, regardless of whether you are a designer, developer, sysadmin, or non-technical contributor.

By integrating with GitHub, all users are able to take advantage of the GitHub flow. Technical users can use the Git and CLI tools they are already familiar with, so we are not limiting them or burdening them with additional overhead. Less technical users can make use of the GitHub flow directly in the browser without needed to install Git, Terraform, or other supporting tooling. This low barrier to entry enables organizations to quickly migrate to version-controlled infrastructure with almost zero prerequisite knowledge.

»Summary

Integrating with GitHub has helped showcase the value of version-controlled infrastructure with Terraform. In addition to providing a single, familiar view where Terraform users can see the status and impact of their changes, the integration also brings about continuous integration and testing for infrastructure changes. The consistent GitHub workflow pairs well with HashiCorp's goals of providing a technology-agnostic workflow for provisioning, securing, and running any infrastructure for any application.

While the integration was not without technical challenges, ultimately those challenges were outweighed by the benefits of familiarity, safety, and trust of our users.

If you are not using Terraform, you can download it today from the Terraform website. If you are already using Terraform, you can start a free trial of Terraform Pro today and start using the GitHub integration!