Managing HashiCorp Consul Access Control Lists (ACLs) with Terraform & Vault

HashiCorp Consul provides a variety of capabilities, including service mesh, security policies through intentions, and a key-value store. For organizations with multiple teams, how do you empower those teams to securely use Consul? In this post, we generate dynamic Consul tokens with HashiCorp Vault and define access control lists (ACLs) as code using the Consul provider for HashiCorp Terraform.

When multiple teams use Consul, it becomes difficult to correlate manually managed policies in Consul with the identity accessing it. For example, you might temporarily add additional access for a developer to edit services, only to forget that someone has write access. A bad actor might be able to obtain the ACL token and use it to access data or change services. The possibility of an insecure ACL policy or long-lived Consul ACL token increases as more users, nodes, and services require access.

To ensure least-privilege access to Consul, you can use HashiCorp Terraform to define and test Consul ACLs and enable audit of policy rules. Then, you can configure the Consul secrets engine for HashiCorp Vault to dynamically generate the API tokens associated with the Consul ACL policy in order to reduce the lifetime of the token and further secure Consul.

»Example Scenario

In this scenario, let’s consider multiple teams. Imagine you are an operator administering to HashiCorp Consul, which development teams use for service mesh and discovery. You enable ACLs on your production Consul cluster and obtain a Consul token with acl = “write” policy to start managing its resources. Besides Consul, your organization uses HashiCorp Vault to manage its secrets, Terraform to manage infrastructure, and Terraform Cloud to store state and test the configuration.

As an operator, you know that different teams need to be able to access Consul but not all of them need access to everything. A new app team needs to be able to view keys to debug consul-template and their connection to other applications, thus requiring access to read any keys related to the app team and all Consul intentions (to debug network policy). When the app team onboards, they should be able to use your organization’s Vault instance to retrieve a token associated with their team.

Let’s see how we can accomplish this by applying Vault and Terraform to configuring Consul ACLs.

»Set up Vault Access to Consul with Terraform

You will need a Consul token to allow Terraform enough access to configure Consul ACLs. The policy associated with the token must have at least an acl = “write” rule.

First, define your Consul address as part of the Terraform provider configuration. The example uses a local Consul instance and scopes the provider to Consul datacenter dc1.

Set the CONSUL_HTTP_TOKEN environment variable to the Consul ACL token. The Consul provider for Terraform will use the management token specified in the variable.

$ export CONSUL_HTTP_TOKEN=aaaaaaaa-bbbb-cccc-dddd-eeeeeeeeeeee

Next, you need to create a Consul policy and token for Vault. The Consul secrets engine for Vault specifically requires a management token. We attach the token to the global-management policy, which provides unlimited access across Consul. For larger Consul deployments, the management token should be scoped to the datacenter.

To create the policy, plan and apply the Terraform configuration.

$ terraform init
 
$ terraform plan
...
Plan: 1 to add, 0 to change, 0 to destroy.
 
$ terraform apply
consul_acl_token.vault: Creating...
consul_acl_token.vault: Creation complete after 0s
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

If you examine the ACL tokens in the Consul UI, you find that Terraform added the Consul ACL token for Vault.

»Configure Consul Secrets Engine for Vault

After creating the Consul ACL token for Vault, use the Vault provider for Terraform to configure HashiCorp Vault with the Consul secrets engine. By enabling the Consul secrets engine, you allow Vault to issue dynamic ACL tokens and attach them to a policy.

First, add the Vault provider to providers.tf with the address of the Vault instance. The example uses a local instance of Vault.

Set the VAULT_TOKEN environment variable to the Vault token with least privilege access to manage secrets engines (create, read, update, delete, list, and sudo to the sys/mounts/* Vault API). The Vault provider for Terraform will use the token specified in the variable.

$ export VAULT_TOKEN=s.bbbbbbbbbbbbbbbbbbbbbb

Next, configure the Consul secrets engine in Vault. Use the consul_acl_token_secret_id Terraform data source to retrieves the secret of the Consul ACL token for Vault. While you can issue a management token for the Consul secrets engine manually, creating it with Terraform allows you to manage and revoke it more dynamically than through the CLI.

When using this data source, the Consul token will be reflected in Terraform state. For security concerns, you should [treat Terraform state as sensitive data]((https://www.terraform.io/docs/state/sensitive-data.html) by encrypting it or storing it in Terraform Cloud. If you do not encrypt state, you should encrypt the Consul token with your own PGP or keybase public key.

The example does not encrypt the token because it uses Terraform Cloud to store and encrypt state.

Pass the token to vault_consul_secret_backend resource. The vault_consul_secret_backend specifies the Consul address for Vault to reference and the lease time for Consul tokens. The example sets the default lease to one hour, which means the Consul tokens expire one hour after issuance. Apply the configuration to create the secrets engine.

$ terraform plan
…
Plan: 1 to add, 0 to change, 0 to destroy.
 
$ terraform apply
...
vault_consul_secret_backend.consul: Creating...
vault_consul_secret_backend.consul: Creation complete after 0s [id=consul]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

You can tell if the Consul secrets engine has been enabled by examining the list of secrets engines and the roles and configuration at the consul/ path in Vault.

$ vault secrets list               
Path          Type         Accessor              Description
----          ----         --------              -----------
consul/       consul       consul_0605a7f4       Manages the Consul backend
 
$ vault read consul/config/access    
Key        Value
---        -----
address    consul:8500
scheme     http

You may require additional automation if you use the Consul secrets engine to issue short-lived tokens for Consul agents or service registration. To rotate tokens for Consul agents, you will need to update the token with the Consul agent API. Similarly, you need to re-register services if you rotate the token associated with the service. In this example, we scope the use of the Consul secrets engine to rotating tokens for reading intentions and keys.

»Define the App Team’s Consul ACL Policies

After creating the Consul management token and configuration for the Consul secrets engine, you can now define the app team’s Consul policies and roles with Terraform and request a dynamic Consul ACL token with Vault. At a minimum, the app team needs to read Consul intentions and keys. You do not want to grant them write access to change any resources at this time.

First, define two policies intended for app team members to view Consul information. The user should be able to view Consul intentions and read from application-related keys. Use the consul_acl_policy resource to create both policies.

For fine-grained access control of intentions based on service, you must include its service destination value. Review Consul documentation for additional information about intention management permissions.

If your organization has Terraform Cloud or Enterprise, you can use HashiCorp Sentinel to check that new policies do not have write access to Consul. This creates a policy check for Consul policies defined as code, enabling teams to self-service access to Consul within certain boundaries. The Sentinel policy only allows teams to update Consul policies with read access.

Each policy will be associated with the app team role created by Vault. Use the vault_consul_secret_backend_role resource to associate both policies to a role labeled app-team. Any token with this role will be able to access Consul based on its associated policies. Apply the configuration with Terraform.

$ terraform plan
…
Plan: 3 to add, 0 to change, 0 to destroy.
 
$ terraform apply
…
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

»Configure App Team Authentication to Vault

In order to allow any app team developer to authenticate to Vault, you configure the GitHub authentication method. An authentication method allows you to specify an identity provider like GitHub to handle application team access to Vault. When you have many teams retrieving various secrets from Vault, an authentication method alleviates the burden of additional configuration.

Add a Terraform configuration that includes the vault_github_auth_backend and vault_github_team resources with a Vault policy limited to read access to on the consul/creds/app-team endpoint.

Apply the configuration.

$ terraform init
 
$ terraform plan
...
Plan: 3 to add, 0 to change, 0 to destroy.
 
$ terraform apply
...
 
Apply complete! Resources: 3 added, 0 changed, 0 destroyed.

Any user under the app team and the example organization in GitHub can now log into Vault with their personal access token. With a read-only Vault policy on their team endpoint to the Consul secrets engine, they will not be able to access other API endpoints in Vault.

»Issue Dynamic Consul ACL Tokens with Vault

An app team user can request a Consul token from Vault by using the consul/creds/app-team endpoint. When you request credentials, Vault generates a new Consul token with a default lease defined by the secrets engine configuration. For example, the token associated with the app-team role must be renewed after one hour. First, the app team logs into Vault with their GitHub token.

$ vault login -method=github token=${GITHUB_TOKEN}
Key                    Value
---                    -----
token                  s.xxxxxxxxxxxxxxxxxxxxxx
token_accessor         iUthJU7SjQQZRAVP0THtvQOu
token_duration         768h
token_renewable        true
token_policies         ["app-team" "default"]
identity_policies      []
policies               ["app-team" "default"]
token_meta_org         example
token_meta_username    some-github-user

Then, they can use their Vault token to request a Consul ACL token.

$ vault read consul/creds/app-team
Key                Value
---                -----
lease_id           consul/creds/app-team/12tQWGMPqmEHAso1ami8D6Pg
lease_duration     1h
lease_renewable    true
accessor           21cd7044-c75a-8801-8c29-9d95959e1e7c
local              false
token              cbbff3b8-c6ad-3db3-dab5-6fdf94df5f97

When the app team uses the token, they can only read intentions and keys. They cannot make updates to intentions or keys or retrieve other information like ACL policies from Consul.

$ consul intention get web db                                        
Source:             web
Destination:        db
Action:             allow
 
$ consul kv get -recurse app/                                 
app/hi/there:
app/toggles:hello
 
$ consul intention create web db                                     
Error creating intention "web => db (allow)": Unexpected response code: 403 (Permission denied)
 
$ consul kv put app/new                                              
Error! Failed writing data: Unexpected response code: 403 (Permission denied)
 
$ consul acl policy list                                             
Failed to retrieve the policy list: Unexpected response code: 403 (Permission denied)

After one hour, the app team will no longer be able to use this token to access intentions and keys.

$ consul intention get web db                                        
Error: Unexpected response code: 403 (ACL not found)
 
$ consul kv get -recurse app/                                        
Error querying Consul agent: Unexpected response code: 403

To regain access to Consul, the app team developer must either renew the lease with vault lease renew consul/creds/app-team/<lease id> or generate a new token with Vault. If a team does not want to manually renew tokens used by service accounts or automation, they can configure Vault agent to automatically authenticate and renew the tokens.

»Conclusion

Using the Consul and Vault providers for Terraform, you created a management token to enable Vault to issue Consul ACL tokens using the Consul secrets engine. After enabling the Consul secrets engine, you used the Consul provider for Terraform to create policies and attach them to roles in Vault. When they need to debug intentions and test their configuration of consul-template, the app team can log into Vault using their GitHub credentials and request Consul ACL tokens from Vault. Vault will handle the renewal and revocation of the token.

By configuring Vault and Consul with Terraform, you can scale and collaborate on Consul ACL policies to secure the cluster. Changes and updates to the policies will reflect in version control and use infrastructure as code practices to maintain security. The addition of the Consul secrets engine generates ACL tokens on-demand and handles the lifetime of the secret.

Learn more about Consul, Vault, and Terraform with the HashiCorp Learn guides. For detailed Consul security recommendations, refer to the Consul Security Model and the complete ACL Guide. Additional documentation on using Terraform to configure Consul and Vault can be found at the Consul provider and Vault provider. For resources on configuring Vault, check out the Consul secrets engine, the Github auth method, and Vault policy documentation.

Any questions? I've created a community forum thread so I can respond to them. Feel free to reach out there!