Deploy Consul cluster peering locally with Minikube

As you scale Kubernetes, you will often find yourself with multiple clusters running different services. HashiCorp Consul’s service mesh on Kubernetes helps register and manage requests between services from a single point of control. But how do you manage traffic between services across multiple Consul datacenters on different Kubernetes clusters? Cluster peering connects Consul datacenters across multiple Kubernetes clusters using administrative partitions, and offers one point of control to manage traffic across Kubernetes clusters.

This post shows how to use Minikube to create multiple Kubernetes clusters in your local development environment to test Consul’s cluster peering and configure services across clusters. Testing locally allows you to verify configurations for services and the requests between services in Consul service mesh before applying changes to your production clusters. The post also dives into a set of Bash scripts to automate the creation and configuration of Kubernetes and Consul clusters that you can adapt for your own local testing environment.

Note that your local development environment must support Docker and offer sufficient resources to run multiple clusters. You can also use a driver that creates clusters in virtual machines. However, not all operating systems support virtual machine drivers. If you create a virtual machine with VirtualBox, you can attach a host-only network to each cluster and configure MetalLB to use that network.

»Set up a Docker bridge network

By default, Minikube sets up each Kubernetes cluster on its own Docker bridge network. This allows clusters to communicate with each other but does not reflect the scenario of external access to a Kubernetes service through a load balancer. Attach each Minikube virtual machine to a second bridge network. The load balancers run in the second network to simulate mesh gateways communicating over a public network..

Create a network named metallb using Docker, with an IP address range of 172.28.0.0/16. Restrict the IP range for Minikube clusters to 172.28.0.0/24 to ensure you do not have overlapping IP addresses for each load balancer:

Each load balancer in each cluster must have its own IP address range so they can route to each other.

»Start Minikube with MetalLB

Consul’s cluster peering requires a mesh gateway to facilitate communication between datacenters on different Kubernetes clusters. The mesh gateways require a load balancer to facilitate inbound and outbound requests to services in each Kubernetes cluster. For the load balancer, this example uses MetalLB, a Minikube addon, which creates a bare-metal load balancer for your cluster.

Start Minikube with the MetalLB addon and set a profile to indicate a specific Kubernetes cluster, in this example dc1:

Connect the cluster to the metallb network:

After the Kubernetes cluster starts, configure the ConfigMap for MetalLB under addresses with an IP address range in the subnet you allocated to the Docker network. For example, you can set the third octet of the network to the number of the cluster:

MetalLB will use IP addresses from this range when you request a Kubernetes service with a load balancer. You will need to configure each Kubernetes cluster with its own profile and update MetalLB’s ConfigMap with an IP address range from the metallb network.

»Install Consul and mesh gateways

Each Kubernetes cluster has its own Consul cluster and datacenter. Update the Helm values with a Consul datacenter to match the Minikube profile name. Set up TLS and enable mesh gateways for Consul cluster peering. By default, the Helm chart will create a Kubernetes service with a LoadBalancer type for the mesh gateway. Install Consul to the consul namespace:

After Consul runs, configure the mesh to peer through mesh gateways:

Update Consul’s proxies to access services through their local mesh gateway:

Repeat these configurations for each Kubernetes cluster you create through Minikube.

»Set up cluster peering

Peer only the clusters that have services communicating with each other in production to maintain least-privilege. The script included in this post peers all clusters to each other for testing purposes. This example peers dc1 to dc2.

Configure dc1 as the peering acceptor, which creates a peering token and sets the target peer to dc2:

Extract the peering token and copy it to dc2:

Set up dc2 as the peering dialer, which initiates the peering request to dc1:

Create a unique peering token for each pair of peered clusters.

»Connecting services

For services to access other services in a peered cluster, export the service from one cluster to another:

Add an intention to allow a service in the peered cluster (dc1) to access the service in the target cluster (dc2):

»Next steps for Consul cluster peering

Once you set up a local testing environment for cluster peering, you can run integration tests for services that need to connect across multiple Consul datacenters and Kubernetes clusters. For a fully automated local testing setup, check out my repository to configure Minikube and Consul across three clusters. Learn more about deploying in our cluster peering for Kubernetes r documentation and deploy an example to Kubernetes clusters on a cloud provider with our tutorial: Connect services between Consul datacenters with cluster peering.