Managing HCS with Terraform Cloud

Recently, we announced the general availability of HashiCorp Consul Service (HCS) on Azure, our first fully-managed service for cloud networking automation. Customers can use HCS to discover and automate connections between services running in both Azure Kubernetes Service (AKS) clusters and Azure VMs without worrying about the operational burden of managing their own Consul deployment. HCS can be launched directly from the Azure Portal, but in this blog, we’re going to show you how to manage HCS using Terraform Cloud.

»Why Use Terraform Cloud instead of the Azure Portal?

Terraform has become a de facto tool for many infrastructure workflows and Terraform Cloud has emerged as a powerful way for organizations to enable greater collaboration for their operators. While one of the major benefits of HCS is its integration within the Azure environment, we know that many customers have spent a lot of time and effort building out a robust Terraform workflow. We don’t want to make customers feel like their only options are to either change the way they are managing their environments or self-host a Consul deployment. Terraform Cloud can be used to manage both the deployment of HCS and services that are interacting with it. Additionally, Terraform Cloud enables multiple operators to interact with the HCS environment while layering on additional guardrails to ensure that changes are being applied in a consistent and compliant manner.

»Getting Started with TFC and HCS

In order to get started, we’ll want to set up a git repository that we can use to integrate with Terraform Cloud. This blog expects that you have created a repository in GitHub to hold your Terraform configurations, and cloned the empty repository locally to work out of. Using this pattern, we will commit our infrastructure as code changes into a shared repository that Terraform Cloud will then read, and execute against. The repository I used as part of this blog can be seen here.

Before we move into configuring the Terraform provider for Consul, we’ll need to make sure that we’ve configured our workspace in Terraform Cloud. This blog assumes that you already have a Terraform Cloud account, have created an organization, and linked your GitHub account to Terraform Cloud.

Once you have logged into your organization, you can select the “New Workspace” icon to get started creating your workspace for HCS. From within the “New Workspace” section, you’ll be presented with several options around how your workflow can execute. In our case, we’re going to be using the version control workflow, which allows us to store our code in git and trigger our workflow off of commits, pull requests, or merges.

On the next screen, we’ll select GitHub as our VCS provider, and then select the repository we plan to store our code in. For this blog, I selected my hcs-consul repository and proceeded to the next step. Finally, we’re given the option to name our workspace which I left as the default (the repository name) and finish up by selecting “Create Workspace”.

From here, Terraform Cloud will configure the link to the GitHub repository. After a few moments you’ll be given the option to configure variables for your workspace.

»Configuring Terraform Cloud Variables for HCS on Azure

We need to configure a few variables that will tell Terraform Cloud how it can interact with HCS on Azure. You can see a screenshot below the variables I’m using in my environment:

Here are the variables being used in this demo: Cluster - the address for my HCS Consul endpoint. In my case, I’m using an externally exposed endpoint to keep this simple. Using a private endpoint is possible as well, but that's a story for another time… Token - HCS on Azure provisions with Consul secured by default. This means that ACLs are enabled out of the box. This token variable is the ACL token that we receive when we configured our cluster and is required in order to interact with Consul Consuldc - This is the name of our Consul datacenter that was configured during HCS provisioning

With these variables in place, we can get started configuring our actual Terraform code

»Configuring the Consul Terraform Provider

We’ll be using the Consul Terraform provider to interact with our HCS on Azure cluster. We’ll want to create 2 files in our empty repository to get started:

• main.tf
• variables.tf

Within the variables.tf file we are going to configure the variables that we also configured within Terraform Cloud. Drop in the following configuration

variable "cluster" {
      description = "variable for Consul Cluster"
      default = "127.0.0.1:8500"
  }

  variable "consuldc" {
      description = "variable for the Consul Cluster Datacenter"
      default = "dc01"
  }

  variable "token" {
      description = "ACL token for HCS cluster"
  }

As you can see, we configure 3 different variables that match the same variables from TFC.

Within our main.tf file, place the following configuration

  provider "consul" {
    address    = var.cluster
    datacenter = var.consuldc
    token      = var.token
  }

This configuration enables the Terraform provider for Consul. With these configurations in place we can add these files and commit them to our Git repository.

git add *
git commit -m “initial file commit”
git push

If you quickly switch back into Terraform Cloud, you should see our run begin, entering the planning stage.

In my environment, I’ve enabled the “auto-apply” functionality to bypass the confirmation requirement on application. This setting is not recommended for production workloads, but it works well for a demo!

After a few moments, we should see a successful application of our plan.

This plan is obviously very basic as all it does is enable the Consul provider. Let’s get started with something a little more interesting!

»Deploying a Workload into HCS on Azure

If we take a look at our HCS on Azure environment, we can see that I’ve deployed a service on our Azure Kubernetes Service cluster that’s been joined.

We can see from Consul that our 3 tier application is currently connected between the tiers via Consul’s service mesh. When we browse our application, however, we can see our application is not connecting

This failure to connect is because HCS on Azure deploys in a secure configuration by default, so with ACL’s enabled, communication is denied by default. What this means is that with no intentions set within Consul to enable connectivity, the connection between tiers will not be successful.

Let’s use our Terraform Cloud configuration to setup our intentions using the Consul Terraform provider.

»Applying Intentions in Consul with Terraform

Update our main.tf file with to match the following code

provider "consul" {
    address    = var.cluster
    datacenter = var.consuldc
    token      = var.token
  }

  resource "consul_intention" "api-allow" {
    source_name      = "frontend"
    destination_name = "api"
    action           = "allow"
  }

This block of Terraform code adds an intention to HCS that allows applications in the frontend service to communicate with the api service. Commit these changes to our GitHub repository with the following command

git add main.tf
git commit -m “adding frontend to api intention” 
git push 

If we quickly switch back into Terraform Cloud we can observe that Terraform Cloud has seen our change committed into our GitHub repository and executed our Terraform manifest against HCS on Azure

If we switch back into HCS, and take a look at our Intention tab we can see that our Frontend to API intention is there.

And finally, if we take a look at our application again, we can see that our Frontend to API communication is successfully connecting now

Our database is still failing. As you might guess, we are missing an intention allowing API to database connectivity lets make another change to our application to enable this intention as well.

Update our main.tf file again to match the below code

provider "consul" {
    address    = var.cluster
    datacenter = var.consuldc
    token      = var.token
  }

  resource "consul_intention" "api-allow" {
    source_name      = "frontend"
    destination_name = "api"
    action           = "allow"
  }

 resource "consul_intention" "db-allow" {
     source_name      = "api"
     destination_name = "db"
     action           = "allow"
   }

As you can see, we’ve added an intention resource for our source API service to our destination DB service. Once again, let’s commit this change to our GitHub repository and observe the results

git add main.tf
git commit -m “adding database intention”
git push 

If we jump back into our HCS on Azure user interface, we can see that we have a new intention listed for API to DB connectivity

And if we switch back into our application we can see both tiers of our application are now connecting successfully

This example is admittedly very basic, but it illustrates nicely how you can combine the governance and execution capabilities of Terraform Cloud alongside Consul’s service mesh capabilities. This overview was only intended to scratch the service of using Terraform Cloud with Consul, and there are a number of additional capabilities we can look into next including things like

• Configuring Namespaces and updating our intentions to span across namespaces
• Configure external service registration in Consul
• Configure Ingress and Terminating Gateways
• Configure Layer 7 Traffic policies

All using the Terraform provider. These infrastructure as code definitions can then be stored alongside your application code to become a definition of how the network policies in your Consul environment are configured.

»Next Steps

Now that we’ve shown you how to get started with HCS using Terraform Cloud and managing new or existing services, it’s your turn to try it out for yourself. HCS offers an on-demand option to get started with, available on the Azure Marketplace. This on-demand option is for a development cluster and should be used for proof of concept purposes only. A production level cluster will be made available as an on-demand option later this year. If you are attending HashiConf Digital, we will be offering a session specifically on HCS. For more information about Consul, please visit our product page.