Introducing HashiCorp Nomad 0.12's New Consul Ingress Gateway Capability

Recently we announced that Nomad now supports running Consul Connect ingress gateways. For the past year, Nomad has been incrementally improving its first-class integration with Consul’s service mesh. Whether through the use of sidecar proxies like Envoy or by embedding the Connect native client library, Nomad supports running tasks that can communicate with other components of a Consul service mesh quickly and securely. Now with support for Consul ingress gateways, Nomad users are able to provide access to Connect-enabled services from outside the service mesh.

»Overview

Ingress gateways enable ingress traffic from services running outside of the Consul service mesh to services inside the mesh. An ingress gateway is a special type of proxy that is registered into Consul as a service with its kind set to ingress-gateway. They provide a dedicated entry point for outside traffic and apply the proper traffic management policies for how requests to mesh services are handled. With this latest release, Nomad can now be used to not only deploy ingress gateway proxy tasks, but configure them too.

Using this feature, Nomad job authors can enable applications external to the Consul service mesh to access those Connect-enabled services. The ingress gateway configuration enables defining one or more listeners that map to a set of backing services.

»Service Configuration

There is a new gateway parameter available for services with a Connect stanza defined at the group level in a Nomad job specification. Inside this stanza are parameters for configuring the underlying Envoy proxy as well as the configuration entry that is used to establish the gateway configuration in Consul.

service {
  gateway {
    proxy {
      // envoy proxy configuration 
      // https://www.nomadproject.io/docs/job-specification/gateway#proxy-parameters
    }

    ingress {
      // consul configuration entry 
      // https://www.nomadproject.io/docs/job-specification/gateway#ingress-parameters
    }
  }
}

The proxy stanza is used to define configuration regarding the underlying Envoy proxy that Nomad will run as a task in the same task group as the service definition. This configuration becomes part of the service registration for the service registered on behalf of the ingress gateway.

The ingress stanza represents the ingress-gateway configuration entry that Consul uses to manage the proxy's listeners. A listener declares the port, protocol (tcp or http), and each Consul service which may respond to incoming requests. When listening on http, a service may be configured with a list of hosts that specify which requests will match the service.

ingress {
  listener {
    port     = 8080
    protocol = "tcp"
    service {
      name = "uuid-api"
    }
  }
}

If the task group containing the ingress gateway definition is configured for bridge networking, Nomad will automatically reconfigure the proxy options to work from inside the group's network namespace for the defined listeners. E.g.,

envoy_gateway_no_default_bind = true
envoy_gateway_bind_addresses "uuid-api" {
  address = "0.0.0.0"
  port    = 8080
}

»Task

Nomad and Consul leverage Envoy as the underlying proxy implementation for ingress gateways. The Nomad task group that defines the ingress service does not require any tasks to be defined - Nomad will derive the task from the service configuration and inject the task into the task group automatically during job creation.

»Discover

To enable easier service discovery, Consul provides a new DNS subdomain for each service fronted by an ingress gateway. To find ingress-enabled services, use:

<service>.ingress.<domain>

By default, <domain> is simply consul. To test that an ingress-gateway is working, the dig command can be used to look up the DNS entry of a service from Consul directly, e.g.

dig @127.0.0.1 -p 8600 <service>.ingress.consul SRV

»Example

The following job specification demonstrates using an ingress gateway as a method of plain HTTP ingress for our UUID generator API Connect-native sample service. This open source example is designed to be runnable if Consul, Nomad, and Docker are already configured.

job "ig-bridge-demo" {

  datacenters = ["dc1"]

  # This group will have a task providing the ingress gateway automatically
  # created by Nomad. The ingress gateway is based on the Envoy proxy being
  # managed by the docker driver.
  group "ingress-group" {

    network {
      mode = "bridge"

      # This example will enable tcp traffic to access the uuid-api connect
      # native example service by going through the ingress gateway on port 8080.
      # The communication between the ingress gateway and the upstream service occurs
      # through the mTLS protected service mesh.
      port "api" {
        static = 8080
        to     = 8080
      }
    }

    service {
      name = "my-ingress-service"
      port = "8080"

      connect {
        gateway {

          # Consul gateway [envoy] proxy options.
          proxy {
            # The following options are automatically set by Nomad if not
            # explicitly configured when using bridge networking.
            #
            # envoy_gateway_no_default_bind = true
            # envoy_gateway_bind_addresses "uuid-api" {
            #   address = "0.0.0.0"
            #   port    = <associated listener.port>
            # }
            #
            # Additional options are documented at
            # https://www.nomadproject.io/docs/job-specification/gateway#proxy-parameters
          }

          # Consul Ingress Gateway Configuration Entry.
          ingress {
            # Nomad will automatically manage the Configuration Entry in Consul
            # given the parameters in the ingress block.
            #
            # Additional options are documented at
            # https://www.nomadproject.io/docs/job-specification/gateway#ingress-parameters
            listener {
              port     = 8080
              protocol = "tcp"
              service {
                name = "uuid-api"
              }
            }
          }
        }
      }
    }
  }

  # The UUID generator from the Connect-native demo is used as an example service.
  # The ingress gateway above makes access to the service possible over tcp port 8080.
  group "generator" {
    network {
      mode = "host"
      port "api" {}
    }

    service {
      name = "uuid-api"
      port = "${NOMAD_PORT_api}"

      connect {
        native = true
      }
    }

    task "generate" {
      driver = "docker"

      config {
        image        = "hashicorpnomad/uuid-api:v3"
        network_mode = "host"
      }

      env {
        BIND = "0.0.0.0"
        PORT = "${NOMAD_PORT_api}"
      }
    }
  }
}

You can run this example by saving it as ingress-gateway.nomad and running the commands,

consul agent -dev
sudo nomad agent -dev-connect
nomad job run ingress-gateway.nomad

Once running, the ingress gateway will be available on port 8080 of the node that the ingress gateway service is running on. The UUID generator service will be listening to a dynamically allocated port chosen by Nomad. Because the UUID generator service is in the Connect service mesh, it will not be possible to connect to it directly as it will reject any connection without a valid mTLS certificate.

In most environments Consul DNS will be configured so that applications can easily discover Consul services. We can use curl and dig to simulate what an application accessing a service through the ingress gateway would look like.

$ curl $(dig +short @127.0.0.1 -p 8600 uuid-api.ingress.dc1.consul. ANY):8080
c8bfae29-3683-4b19-89dd-fbfbe691a6e7

»Limitations

At the moment, Envoy is the only proxy implementation that can be used by Nomad and Consul as an ingress gateway. When being used as an ingress gateway, Nomad will launch Envoy using the docker task driver, as there is not yet support for manually specifying the proxy task.

Ingress gateways are configured using Consul configuration entries, which are global in scope across federated Consul clusters. When multiple Nomad regions define an ingress gateway under a particular service name, each region will rewrite the ingress-gateway Configuration Entry in Consul for that service. In practice, typical individual ingress gateway service definitions would be the same across Nomad regions, causing the extra writes to turn into no-ops.

When running the ingress gateway in host-networking mode, the Envoy proxy creates a default administration HTTP listener that is bound to localhost. There is no way to disable or secure the Envoy administration listener envoy/2763. Any other process able to connect to localhost on the host machine will be able to access the Envoy configuration through its administration listener, including Service Identity Tokens for the proxy when Consul ACLs are in use.

»Conclusion

In this blog post, we shared an overview of Nomad's Consul ingress gateway integration and how it can be used to configure and run Consul ingress gateways on Nomad. Using this integration, job specification authors can easily create endpoints for external applications to make requests to Connect-enabled services that would otherwise be accessible only through the Consul service mesh. For more information about ingress gateways, see the Consul ingress gateway documentation.

For more information about Nomad, please visit our docs