Proxy Ingress to Consul Service Mesh

Consul service mesh delivered through Consul Connect provides service-to-service connection authorization and encryption using mutual Transport Layer Security (mTLS). A secure mesh network, such as Connect, requires ingress points to act as gateways to enable external traffic to communicate with the internal services. While there are various technological specific resources for this task, sometimes you need or want the flexibility of a traditional reverse proxy server. In these cases, you can use other 3rd-party proxies with consul-template to create a robust ingress proxy for your mesh network. Note, you must use a proxy that supports mTLS.

Two reliable and popular proxies that support mTLS are NGINX and HAProxy. Each of them can be configured using consul-template to work as a native ingress proxy. This can be beneficial when performance is of utmost importance.

Below you can walk through a complete example setup for each proxy. The examples use consul-template to dynamically generate the proxy configuration and the required certificates for the proxy to communicate directly with the services inside the mesh network.

This blog post is only meant to demonstrate features and many things have been simplified. It is not a production ready or secure deployment. Each of the services are configured for demonstration purposes and will run in the foreground and output its logs to that console. They are all designed to run from files in the same directory. In this blog post, we use the term ingress proxy to describe Nginx or HAProxy. When we use the term sidecar proxy, we mean the Consul Connect proxy for the internal service.

»Common Infrastructure

The example setup below requires Consul, NGINX, HAProxy, and Python. The former is available at the link provided, the latter 3 should be easily installable with the package manager on any Linux system.

Both ingress proxies need a service to proxy to, for that you need Consul, a service (E.g. a simple webserver), and the Connect sidecar proxy to connect it to the mesh. The ingress proxy will also need the certificates to make the mTLS connection.

»Start Consul

Connect requires Consul 1.2.0 or newer. First start your agent.

$ consul agent -dev

»Start a Webserver

For the webserver you’ll use Python’s simple built-in server included in most Linux distributions and Macs.

$ python -m SimpleHTTPServer

Python’s webserver will listen on port 8000 and publish an index.html by default. For demo purposes, create an index.html for it to publish.

$ echo "Hello from inside the mesh." > index.html

Next, you’ll need to register your “webserver” with Consul. Note, the Connect option registers a sidecar proxy with the service.

$ echo '{
  "service": {
    "name": "webserver",
    "connect": { "sidecarservice": {} },
    "port": 8000
  }
}' > webserver.json

$ consul services register webserver.json

You also need to start the sidecar.

$ consul connect proxy -sidecar-for webserver

Note, the service is using the built-in sidecar proxy. In production you would probably want to consider using Envoy instead.

»Create the Certificate File Templates

To establish a connection with the mesh network, you’ll need to use consul-template to fetch the CA root certificate from the Consul servers as well as the applications leaf certificates, which Consul will generate.

You need to create the templates that consul-template will use to generate the certificate files needed for each of the ingress proxies. The template functions caRoots and caLeaf require consul-template version 0.23.0 or newer.

Note, the names, “nginx” and “haproxy”, used in the leaf certificate templates. They need to match the names used to register the ingress proxy services with Consul below.

»ca.crt

Both HAProxy and NGINX require the CA certificate.

$ echo '{{range caRoots}}{{.RootCertPEM}}{{end}}' > ca.crt.tmpl

»Nginx cert.pem and cert.key

NGINX requires the certificate and key to be in separate files.

$ echo '{{with caLeaf "nginx"}}{{.CertPEM}}{{end}}' > cert.pem.tmpl

$ echo '{{with caLeaf "nginx"}}{{.PrivateKeyPEM}}{{end}}' > cert.key.tmpl

»HAProxy certs.pem

HAProxy wants the leaf certificate and key in the same file.

$ echo '{{with caLeaf "haproxy"}}{{.PrivateKeyPEM}}{{.CertPEM}}{{end}}' > certs.pem.tmpl

»Setup The Ingress Proxies

Each ingress proxy will need to be registered with Consul and have a templated configuration file. In each of the templated configuration files, the connect function is called and returns a list of the services with the passed name “webserver” (matching the registered service above). The list Consul creates is used to create the list of back-end servers to which the ingress proxy connections.

The ports are set to different values so you can have both proxies running at the same time.

»NGINX

First, register the nginx ingress proxy service with the Consul servers.

$ echo '{
  "service": {
    "name": "nginx",
    "port": 8081
  }
}' > nginx-service.json

$ consul services register nginx-service.json

Next, configure the NGINX configuration file template. Set it to listen on the registered port and route to the Connect-enabled servers retrieved by the Connect call.

»nginx-proxy.conf.tmpl

$ cat > nginx-proxy.conf.tmpl << EOF
daemon off;
master_process off;
pid nginx.pid;
error_log /dev/stdout;

events {}

http {
  access_log /dev/stdout;

  server {
    listen 8081 defaultserver;

    location / {
{{range connect "webserver"}}
      proxy_pass https://{{.Address}}:{{.Port}};
{{end}}
      # these refer to files written by templates above
      proxy_ssl_certificate cert.pem;
      proxy_ssl_certificate_key cert.key;
      proxy_ssl_trusted_certificate ca.crt;
    }
  }
}
EOF

»HAProxy

Just like above, you first should register the haproxy ingress proxy service with the Consul servers with the port it will use.

$ echo '{
  "service": {
    "name": "haproxy",
    "port": 8082
  }
}' > haproxy-service.json

$ consul services register haproxy-service.json

Next, configure the HAProxy configuration file template. Set it to bind to the registered port and routing to the Connect-enabled backend services.

»haproxy.conf.tmpl

$ cat > haproxy.conf.tmpl << EOF
defaults
	mode	http
    timeout connect 5000
    timeout client  50000
    timeout server  50000

backend connect
{{range connect "webserver"}}
    server webserver {{.Address}}:{{.Port}} ssl verify required ca-file ./ca.crt crt ./certs.pem
{{end}}

frontend app
    bind :8082
    default_backend connect
EOF

»Consul Template

The final piece of the puzzle, tying things together are the consul-template configuration files! These are written in HCL, the Hashicorp Configuration Language, and lay out the commands used to run the proxy, the template files, and their destination files.

»nginx-ingress-config.hcl

$ cat > nginx-ingress-config.hcl << EOF
exec {
  command = "/usr/sbin/nginx -p . -c nginx-proxy.conf"
}
template {
  source = "ca.crt.tmpl"
  destination = "ca.crt"
}
template {
  source = "cert.pem.tmpl"
  destination = "cert.pem"
}
template {
  source = "cert.key.tmpl"
  destination = "cert.key"
}
template {
  source = "nginx-proxy.conf.tmpl"
  destination = "nginx-proxy.conf"
}
EOF

»haproxy-ingress-config.hcl

$ cat > haproxy-ingress-config.hcl << EOF
exec {
  command = "/usr/sbin/haproxy -f haproxy.conf"
}
template {
  source = "ca.crt.tmpl"
  destination = "ca.crt"
}
template {
  source = "certs.pem.tmpl"
  destination = "certs.pem"
}
template {
  source = "haproxy.conf.tmpl"
  destination = "haproxy.conf"
}
EOF

»Running and Testing

You are now ready to run the consul-template managed ingress proxies. When you run consul-template, it will process each of the templates, fetching the certificate and server information from consul as needed, and render them to their destination files on disk. Once all the templates have been successfully rendered it will run the command starting the proxies.

Run the NGINX managing consul-template instance.

$ consul-template -config nginx-ingress-config.hcl

Run the HAProxy managing consul-template process.

$ consul-template -config haproxy-ingress-config.hcl

Now with everything running, you are finally ready to test the proxies.

#NGINX 
$ curl http://localhost:8081
Hello from inside the mesh!

#HAProxy
$ curl http://localhost:8082
Hello from inside the mesh!

»Conclusion

In this blog post you walked through setting up both NGINX and HAProxy to work as proxies to provide ingress to services contained in a Consul Connect service mesh. This can work for any ingress proxy that supports mTLS and uses a configuration file. And while we’ll be building other, simpler ways to do this in the future, these standard proxies will continue to be a great way to get the flexibility you need to integrate with your existing infrastructure.

For more details, the Learn Guides are a great resource. From picking up the basics in the Service Mesh introductory guide to setting up Consul in Production. You might also want to keep the Consul Connect reference documentation handy.

You can also read more about consul-template, like using it to configure 3rd party load balancers, or look at the documentation kept with the source code in the git repository.