Enabling Service Discovery Across Hybrid Infrastructure Environments using HashiCorp Consul and AWS Cloud Map

Update: On March 27, 2019 AWS Announced General Availability of AWS App Mesh. HashiCorp Consul supports service discovery between App Mesh and Consul through this integration with Cloud Map. Read this blog to learn more about how this integration works.

Yesterday, AWS announced general availability of their new service discovery tool, AWS Cloud Map. AWS users will now have the capability to discover resources and services within AWS environments, conduct health checks, and integrate with other service mesh offerings, like HashiCorp Consul. HashiCorp, an Advanced tier member of the AWS Partner Network, worked closely with AWS engineers and is pleased to announce that HashiCorp Consul is offering launch-day support of AWS Cloud Map.

Consul is a distributed service mesh to connect, secure, and configure services across any runtime platform and public or private cloud. With the launch day support of AWS Cloud Map, Consul users can seamlessly discover services registered in AWS and vice versa. Additionally, having a central catalog gives users a unified view of services, regardless of whether those services are in the cloud or on-premises.

»AWS Cloud Map to Consul

Services registered in AWS Cloud Map are actively synced to the Consul catalog without having to make any changes to existing Cloud Map services themselves. Consul users can then discover and connect to these services using Consul DNS interface or the HTTP API.

The syncing process by default syncs all services registered in AWS Cloud Map to Consul. Below is an example of a service called “web” registered in AWS Cloud Map.

The service has three service instances registered with A name records. Shortly after registering the service in AWS Cloud Map, it becomes available in Consul and can be queried using Consul’s DNS interface.

# From a Non-AWS or On-premise server
$ dig web.service.consul
…..
;; QUESTION SECTION:
;web.service.consul.		IN	A

;; ANSWER SECTION:
web.service.consul.	0	IN	A	10.0.0.3
web.service.consul.	0	IN	A	10.0.0.2
web.service.consul.	0	IN	A	10.0.0.1

;; ADDITIONAL SECTION:
web.service.consul.	0	IN	TXT	"external-source=aws"
web.service.consul.	0	IN	TXT	"external-source=aws"
web.service.consul.	0	IN	TXT	"external-source=aws"

The same service can be accessed using Consul’s HTTP API.

Externally registered services from AWS Cloud Map are clearly marked in the UI with the AWS icon and tag (Consul 1.4.1+ is required for the icon to be visible, but earlier versions of Consul work for the sync functionality):

»Consul to AWS Cloud Map

Services registered in the Consul catalog can be actively synced into AWS Cloud Map as well. AWS users and applications can discover those services using one of the AWS SDKs, API calls, or DNS queries. This enables users and applications in AWS to extend service discovery capabilities beyond AWS infrastructure. For example, an application running in AWS can discover a database running in an on-premises data center.

The syncing process by default syncs all services registered in Consul to AWS Cloud Map. In this release, services can also be filtered on their name prefix by supplying the -aws-service-prefix argument. Below is an example of a service called “db” registered in Consul and can be seen using the UI or DNS interface.

$ dig db.service.consul SRV
…..
;; QUESTION SECTION:
;db.service.consul.		IN	SRV

;; ANSWER SECTION:
db.service.consul.	0	IN	SRV	1 1 6379 0a000001.addr.dc1.consul.
db.service.consul.	0	IN	SRV	1 1 6380 0a000001.addr.dc1.consul.

Shortly after registering the service in Consul it becomes available in AWS Cloud Map as shown below.

»Catalog Syncer

The sync process can be run in an environment with access to AWS and Consul API. There are additional options that can be set to configure sync direction, service name prefix, and more.

$ ./consul-aws sync-catalog -h
Usage: consul-aws sync-catalog [options]

  Sync AWS services, and more with the Consul service catalog.
  This enables AWS services to discover and communicate with external
  services, and allows external services to discover and communicate with
  AWS services.
…..

Command Options


  -aws-dns-ttl=<int>
     DNS TTL for services created in AWS CloudMap in seconds. (Defaults
     to 60)

  -aws-namespace-id=<string>
     The AWS namespace to sync with Consul services.


  -aws-pull-interval=<string>
     The interval between fetching from AWS CloudMap. Accepts a sequence of decimal numbers, each with optional fraction and a unit suffix,
     such as "300ms", "10s", "1.5m". Defaults to 30s)

  -aws-service-prefix=<string>
     A prefix to prepend to all services written to AWS from Consul. If
     this is not set then services will have no prefix.

  -consul-service-prefix=<string>
     A prefix to prepend to all services written to Consul from AWS. If
     this is not set then services will have no prefix.

  -to-aws
     If true, Consul services will be synced to AWS.

  -to-consul
     If true, AWS services will be synced to Consul.

The sync functionality is part of the consul-aws project. This project ships as a single binary that can be run outside or inside the AWS environment. By building this project separate from Consul core, we can iterate on AWS integrations more quickly and release new versions without forcing users to do a full Consul upgrade.

For a full walkthrough for using Consul-AWS, read the guide.

»Extending Consul Catalog to AWS App Mesh

At re:Invent, AWS also announced AWS App Mesh, AWS’s service mesh offering for native AWS environments. AWS App Mesh uses AWS Cloud Map as the service registry which allows AWS App Mesh to discover instances of a given service in AWS. Because the integration of AWS Cloud Map and Consul allows services registered using Consul to be available in AWS Cloud Map, AWS App Mesh can also discover these services. This lets users extend AWS App Mesh to private clouds and beyond.

»Conclusion

The AWS Cloud Map and HashiCorp Consul integration enables users to extend service discovery to heterogeneous environments without requiring a large amount of manual configuration. Organizations utilizing hybrid cloud infrastructure or operating in multiple regions can ensure that their services have access to necessary resources across all of their data centers. Because of this integration, users will now have a seamless service discovery experience for both native and non-native AWS environments that can easily be incorporated into their existing workflows. Those operating within AWS-only environments can now utilize resources located outside those environments, like on-premises data centers, and Consul users gain greater visibility of services being created by other tools, like AWS App Mesh.

The AWS Cloud Map and Consul integration is available now. To learn more, please refer to the Consul-AWS guide in the Consul documentation.