Consul Connect Native Tasks in HashiCorp Nomad 0.12

With the launch of Nomad 0.12, Nomad now supports running Consul Connect native tasks! Last October, Nomad v0.10 debuted support for running tasks making use of Consul Connect. Through the use of sidecar proxies like Envoy, Nomad tasks were enabled to communicate with other components of a Consul service mesh quickly and securely. Now with support for Connect native, Nomad users are to make use of Connect features directly from their applications.

»Overview

Consul Connect native is a way for applications to join a Consul service mesh without requiring sidecar services. Instead of relying on a sidecar to manage the service mesh networking details, applications are embedded with a library capable of managing connections to other Connect-enabled services. Implementing applications as Connect native services provides a few important benefits.

Using this feature, operators can expect a reduction in overall overhead on cluster-wide resources. While proxies like Envoy aim to be efficient, they can still consume a substantial amount of compute and memory resources. For example, every Connect enabled Nomad task will require its own instance of a sidecar proxy. When spread across an entire cluster, the additional resource consumption really adds up! With Connect native, applications joining the service mesh need only to import a small library - no extra resources required.

Those who compose Nomad jobs will enjoy the simplified task configuration when deploying Connect native tasks. Although Nomad does its best to enable reasonable defaults when using a sidecar, nothing is simpler to configure than not running a sidecar in the first place. Connect native services merely need to indicate their underlying Nomad task. Nomad and Consul take care of the rest.

»Task Configuration

There is a new native parameter available for services with a Connect stanza defined at the group level in a Nomad job specification. When set to true, Nomad understands the configured task is the implementation of the Connect native service. If there is only one task in the group, setting the task parameter is not required.

      connect {
        native = true
      }

Unlike when using sidecar proxies, there is no need to describe the upstreams the task will Connect to in the jobspec. Connect native enables applications to reference Consul DNS names without actually configuring Consul DNS. For example, to connect to a service with the name myapp, the client would make a request using the Connect native library to:

https://myapp.connect.consul/

When communicating with Consul that has TLS and/or ACLs enabled, environment variables can be set on the task to provide necessary information for establishing a secure connection. The Connect library provided by HashiCorp understands the same environment variables familiar to Consul itself, e.g.

• CONSUL_HTTP_ADDR
• CONSUL_HTTP_TOKEN
• CONSUL_CACERT
• CONSUL_CLIENT_CERT
• CONSUL_CLIENT_KEY

»Nomad Client Configuration

When using Consul enabled with TLS, the Nomad client tries to be helpful by optionally sharing its Consul TLS configuration with Connect native tasks. Certificates are copied into the task allocation and the standard Consul environment variables to make use of them are set in the task runtime environment. This behavior can be configured by setting the share_ssl parameter in the Nomad agent consul stanza. Nomad does not share its Consul token when Consul ACLs are enabled - instead, a Consul service identity token is generated automatically on behalf of the Connect native task and supplied via the CONSUL_HTTP_TOKEN environment variable.

      consul {
        share_ssl = true
      }

»Connect native Library

In order to make an application Connect native, it requires the use of a library that automatically handles the Connect service mesh networking. HashiCorp provides a high-quality reference implementation for Go applications. Other languages can implement their own by following the provided guidance in the Connect documentation.

In a nutshell, the Connect library should replace the use of the standard library http.TLSConfig struct for servers, and the http.Client struct for clients. These components will be substituted for Connect-aware alternatives that automate the networking configuration.

The following snippet of example code establishes an HTTPS server that works within the Connect service mesh.

service, _ := connect.NewService("myserver", consulClient)

(&http.Server{
	Addr:      "0.0.0.0",
	TLSConfig: service.ServerTLSConfig(),
	Handler:   handler(),
}).ListenAndServeTLS("", "")

The NewService method instantiates a Service object from which other Connect native components can be derived. The consulClient parameter here is just an instance of the normal Consul HTTP API client. Note that ListenAndServeTLS normally requires the certFile and keyFile parameters to be set. Here, they are left empty because those certificates will be managed automatically by the special TLSConfig struct - which implements TLSConfig.Certificates and TLSConfig.GetCertificate specifically for joining the service mesh.

Likewise, the following snippet of example code establishes an HTTPS client that works over the Connect service mesh.

service, _ := connect.NewService("myclient", consulClient)
client := service.HTTPClient()

From here, the instantiated client is very similar to the standard HTTP client library - except that it works for connecting to Connect enabled services. For example, making a GET request to another Connect service, using the built-in Consul DNS capabilities.

response, _ := client.Get("https://myserver.connect.consul/")

This is all that is necessary for establishing connections between two Connect native applications. Of course, Connect native tasks can also connect to any other Consul service mesh service. It all just works!

»Example

The following job specification contains both a Connect native API service and HTTP frontend service. This open source example is designed to be runnable if Consul, Nomad and Docker are already configured.

	job "cn-demo" {
	  datacenters = ["dc1"]

	  group "generator" {
		network {
		  port "api" {}
		}

		service {
		  name = "uuid-api"
		  port = "${NOMAD_PORT_api}"

		  connect {
		    native = true
		  }
		}

		task "generate" {
		  driver = "docker"

		  config {
		    image        = "hashicorpnomad/uuid-api:v3"
		    network_mode = "host"
		  }

		  env {
		    BIND = "0.0.0.0"
		    PORT = "${NOMAD_PORT_api}"
		  }
		}
	  }

	  group "frontend" {
		network {
		  port "http" {
		    static = 9800
		  }
		}

		service {
		  name = "uuid-fe"
		  port = "9800"

		  connect {
		    native = true
		  }
		}

		task "frontend" {
		  driver = "docker"

		  config {
		    image        = "hashicorpnomad/uuid-fe:v3"
		    network_mode = "host"
		  }

		  env {
		    UPSTREAM = "uuid-api"
		    BIND     = "0.0.0.0"
		    PORT     = "9800"
		  }
		}
	  }
	}

You can run this example by saving it as connect-native.nomad and running the commands,

$ consul agent -dev
$ sudo nomad agent -dev-connect
$ nomad job run connect-native.nomad

Once running, the frontend service will be available on port 9800 of the node the service is running on. Under the hood, the frontend service is connecting to the API generator service to retrieve a new UUID on demand for every request. Each service is Connect native, requiring no sidecars to establish a secure connection.

»Limitations

There are some features enabled by the Envoy sidecar proxy that are not available to Connect native tasks. For example, Consul has the ability to configure the Envoy sidecar to create Expose Paths which allow non-secure traffic to pass through on specific HTTP paths. A common use case for this is to allow Consul HTTP checks to work with Connect tasks. The Connect native equivalent would be to manually establish such HTTP endpoints on a normal, non-secure HTTP server listener.

»Conclusion

In this blog post, we shared an overview of Nomad's Consul Connect native integration and how it can be used to Connect non-sidecar enabled services to interact with a Consul service mesh. Using this integration, job specification authors can run their Connect native tasks on Nomad with little more than an extra line of configuration. For more information, see the Consul Connect native documentation.

For more information about Nomad, please visit our docs