Introducing Federation on HashiCorp Consul Service

In Consul 1.8 we enhanced Consul’s federation capabilities by enabling WAN Federation over mesh gateways. Consul datacenter federation enables operators to extend their Consul environments to include additional datacenters. Federating these environments allows Consul to share security and traffic policies with each other. This process ultimately lowers the operational overhead of connecting applications across distinct regions, as well as enables a more consistent, and improved, security model. Furthermore, organizations can use federation to build additional resiliency into their service mesh and applications. An example of this could be creating a policy that directs traffic to a secondary datacenter in the event that the primary one is unreachable, or for A/B testing.

Today, we are pleased to announce that we have added federation as a preview feature for HCS, enabling users to connect together multiple HCS datacenters. In this blog, we’ll explain what federation is and the benefits it provides to users.

»Recapping Federation

Earlier in this post we discussed some of the benefits of federating Consul workloads but it is also helpful to peel back what we are talking about functionally when it comes to federation, specifically on Consul. On its own, federation refers to facilitating the interconnection of two or more geographically separate computing clouds. The goal is that each of these clouds should have the ability to interact and communicate with one another, even if they are operating independently. We can apply the need for this functionality to Consul by looking at how Consul servers communicate with one another.

Consul utilizes a gossip protocol for sharing information between servers. In the case of service mesh, we layer on the additional need of being able to distribute connectivity information across these clusters in addition to the previously mentioned gossip. In order to connect multiple datacenters together, Consul requires that all agents are connectable over their WAN-advertised address.

For more traditional environments this doesn’t presents a problem, but challenges arise as organizations shift towards more dynamic, microservice-based architectures operating across multiple runtime platforms (i.e. Kubernetes, Virtual Machine, etc.). Issues can include possible overlapping IP addresses, workload isolation requirements, or even simply security concerns about the number of exposed entry points into the network.

With Consul’s federation, we instead designate a gateway for datacenter as the path traffic should follow. This gateway is responsible for providing a communication path between the agents and resources within that specific datacenter and then sharing it with other gateways similarly positioned in the other datacenters participating in the mesh. Once implemented, organizations can have independently operating Consul datacenters that are capable of sharing policy and connecting across multiple environments with a limited number of endpoints that need to be monitored and secured. For more information on WAN federation, please read our documentation.

»Why Federation Matters

So now that we have a stronger understanding of what federation is, the next question is why does it matter? Simplifying the communication between datacenters helps ease the operational burden on organizations, but there are a number of strategic benefits for deploying a federated architecture. Let’s explore a few of those use cases.

»Cross-region Deployments

One of the biggest benefits of cloud is that it enables organizations to deploy applications in multiple regions which can help in improving application performance and reducing latency. However, managing these applications across regions present a new set of challenges that existing networking practices are not necessarily optimized for. One way that organizations can tackle these challenges is to treat each region like an independent network, maintaining them according to organizational standards and applying security best practices in a per region fashion. This isn’t really a scalable solution though, and it detracts from the benefit of cloud: having instant access to multiple locations and being able to deploy workloads to those regions quickly.

By instead leveraging a federated pattern of Consul, enterprises can consistently apply networking and security policies across all applications, regardless of which region they have been deployed in. One question you may be asking, is how does this differ from using Consul without federation? From a functionality standpoint, it’s fairly similar. The main benefit is improving the way these policies are applied when you have applications that live across regions.

Typically, this would result in you accessing a management interface within that region and reapplying the same configuration. For example, let’s say that you have an application deployed in two different Azure regions. I could treat these like two independent datacenters with their own Consul agents. If I decided to make a change to my service policies in Consul, I would need to apply those changes to both regions. At two regions for two applications, this doesn’t seem to add a large burden, but now think about the impact as we extend to more regions and add even more services to manage. By federating these datacenters, I have a way to apply policies and changes more globally.

»Improved Application Resiliency

The next question that might arise, is why would I need to have multiple instances of an application running in multiple regions or clouds? The answer is application resiliency. Another benefit of leveraging Consul federation is the ability to create rules that will redirect traffic to a secondary location in the event that a service in a primary datacenter is unreachable. This enables organizations to be prepared for outages that may fall outside of their control, like regional outages. While adding redundant applications are not unique to a federated Consul model, the improved performance and automated traffic policies are the added benefit.

»Progressive Application Delivery

Another benefit of utilizing a federated architecture is the way it enhances the application deployment lifecycle. As organizations roll out new versions of an application it becomes an increasingly complex process. As application deployment methodologies have matured, we’re seeing that the simple “switch” from version 1 of an application to version 2 isn’t sufficient. Application developers want ways to send a fraction of traffic to new services, and gradually increase the “throttle” as application stability is verified.

Additionally, application teams want this ability to be self-service, and able to be rolled back in the event of a problem. This need creates a bottleneck during deployment without a service mesh, as multiple teams would need to be engaged to facilitate this traffic pattern. The result is that developers are forced to wait on slower, ticket-based processes that hinder velocity. As we’ve discussed in previous blogs and webinars, Consul helps close the gap between networking and application delivery to automate applying traffic policies and enabling progressive delivery tactics like these canary deployments or A/B testing.

However, Consul’s ability to enable progressive delivery does not necessarily require federation, so what is the major benefit? When we start looking at typical cloud environments we find that these deployments stretch globally across multiple regions. The way an operator interacts with policy needs to align to that workflow as well. The policies you apply locally today will span to globally tomorrow as your environment matures.

A practical example of this can be when Consul federation is paired with something like Nomad’s multi-cluster deployment feature. If an organization is leveraging Nomad for application orchestration and utilizing multi-cluster deployments, Consul federation further enhances this capability by applying traffic and security policies at the initial orchestration point. Organizations can now deploy a single application across multiple regions with the proper networking rules in a single workflow.

»Next Steps

Federation between HCS datacenters is now available as a preview feature of the platform. Because it is a preview, it does require additional steps for enabling it with your HCS environment. For more information about how to deploy HCS federation, please refer to our new learn guide. If you are new to HCS and looking to get started, learn how to deploy a datacenter through the Azure portal using your existing Azure account. One important item to note is that this feature currently only supports federation between HCS datacenters, it will not work with self-hosted Consul datacenters. For more information about Consul itself, please visit our product pages.