Using Consul’s transparent proxy on virtual machines

Transparent proxy helps secure services in a service mesh by forcing traffic through a proxy like Envoy to prevent unauthorized direct access to the application. It allows a service to resolve and connect to another service using DNS instead of through proxy upstreams listening on localhost. In HashiCorp Consul, a service can connect to an upstream service using transparent proxy and Consul’s DNS as long as an intention allows traffic between the services.

This post shows you how to configure Consul’s transparent proxy on virtual machines (VMs). Your operating system must support iptables in order for transparent proxy to restrict traffic on the VM. Furthermore, you must run Consul 1.11.1 or later to use transparent proxy. For a full example, check out my Terraform configuration that uses a user data script to set up transparent proxy. While the example uses Terraform to set up Consul with transparent proxy through user data, you can adapt its scripts for your own VM configuration.

»Configure Consul client

Before your service can use transparent proxy, set up a Consul client on your virtual machine with service mesh and DNS forwarding enabled. Make sure Consul runs under the consul Linux user. You will need its user ID to make it easier to exclude Consul-related traffic from transparent proxy redirection.

If you have access control lists (ACLs) enabled, create a Consul ACL token for the node identity. This token includes a policy to write nodes and read services in Consul.

The Consul client configuration sets both agent and default tokens to register the virtual machine and resolve services with Consul for service discovery.

In order to use Consul service discovery on the virtual machine, forward the virtual machine’s DNS to Consul. When you use Consul service discovery, you can configure your service to look up another service in the mesh using its virtual service name. Set up DNS forwarding for your DNS server. For example, add Consul to the systemd-resolved configuration file and restart the process to forward any DNS requests to the service mesh to Consul.

[Resolve]
DNS=127.0.0.1:8600
Domains=~consul

Start the Consul client. The virtual machine should register as a node in Consul. Before registering the service and starting the proxy, create a user for the proxy named envoy. Consul needs this user ID in order to exclude traffic to and from the proxy.

»Register the service

Define a service to add it to the mesh and enable transparent proxy. The service definition sets the service’s name, port, and proxy mode.

Consul assigns a virtual IP in the 240.0.0.0/4 range to a service with transparent proxy enabled. This allows downstream services to connect to upstream services using a unique IP assigned by the service mesh. You can register the service with the Consul API or add the service definition to the Consul configuration directory on your VM. Adding the service definition to the configuration directory registers the service each time the Consul client restarts.

»Redirect traffic to proxies

After installing and configuring Consul and Envoy, redirect traffic to and from the virtual machine using transparent proxy. If you have a mesh configuration entry defined, check if TransparentProxy.MeshDestinationsOnly is true. This setting will deny all inbound and outbound requests to the VM once you redirect traffic. As a result, make sure you install the application and its dependencies before redirecting traffic since transparent proxy will deny requests to download packages by default.

Issue the consul connect redirect-traffic command to enable transparent proxy on the VM for the named service. The command should include the user ID for consul and envoy in order for transparent proxy to exclude traffic from Consul and Envoy from redirection.

Once you issue the command, any connection you have to the VM (including SSH) will disconnect. The command adds iptables rules that will deny existing inbound traffic unless you have specified exclusions. If you need to exclude inbound or outbound traffic from specific ports or IP addresses, append additional arguments specifying exclusions. For example, you may need to exclude inbound traffic to the SSH port for login:

Next, install and run Envoy on the VM. Issue the consul connect envoy command for Consul to run Envoy. If you have ACLs enabled, make sure that you create a service token and pass it to the command using the -token argument.

To automate transparent proxy configuration, you can add a script enabling traffic redirection to the proxy’s system file before you start the proxy. After stopping the proxy, you may want a script to clean up iptables. Use a systemd unit file to use the Consul ACL token from an environment file and redirect traffic before starting the proxy:

Review the example for scripts to set up traffic redirection and clean up iptables. After starting the proxy, the service can now connect to another service in the mesh.

»Connecting services

If you have Consul ACLs enabled, you have a service mesh that denies all traffic between services by default. Make sure you have an intention authorizing communication between your source and destination services. The example uses Terraform to define an intention for the API service that allows inbound traffic from the web service:

Once you define an intention, log into your VM for a downstream service. A service can resolve the in-mesh IP of an upstream service using a DNS hostname in the format of ${service_name}.virtual.consul. For example, the VM hosting the web service resolves the API service’s virtual IP:

The intention allows the web service to connect to the API service:

»Next steps

Once you try transparent proxy on a VM, check out my Terraform module to automate the setup of transparent proxy. The module takes variables and generates cloud-init user data you can pass to your VM to configure a Consul client, proxy, and application on startup. For more on transparent proxy, including its use for Kubernetes services, review the Enable transparent proxy mode documentation.