Products & Technology

Exploring the HashiCorp Consul Ecosystem for Kubernetes

Nov 18 2019Peter McCarron

HashiCorp is a sponsor at this year’s KubeCon North America, happening November 18 - 21 in San Diego. At our booth on the Expo floor (S-27), we’ll have experts to explain how you can use HashiCorp Consul for service discovery, service registration, and service mesh for Kubernetes and non-Kubernetes environments.

At KubeCon Barcelona, we announced our participation in Microsoft’s new Service Mesh Interface (SMI). Our integration enables Kubernetes users to manage HashiCorp Consul service mesh intentions directly through tools like kubectl, helm, or Terraform. That announcement described one of the ways Consul connects with Kubernetes environments. In support of KubeCon San Diego, this blog highlights some of the capabilities that HashiCorp Consul can bring to Kubernetes environments and some of the key integrations from our ecosystem that can be deployed in conjunction with Consul.

»Native Capabilities & Getting Started

Out of the box, Consul supports both standalone Kubernetes and managed Kubernetes (Microsoft AKS, AWS EKS, Google Cloud GKE, and RedHat Openshift) in a number of different ways. Consul agents can run inside Kubernetes Pods, in either server or client mode, as part of a cluster in a pure Kubernetes environment or in heterogeneous environments running across multiple clouds. This makes it easier for organizations to start incorporating Kubernetes into their existing infrastrastructure and manage all their services with a single control plane. Upon installation, Consul can be used as a central service registry, help secure and facilitate service-to-service communication, and enhance observability over network and service health. Tools like Helm or Shipyard can help you get started with Consul on Kubernetes.

Helm is a package manager for Kubernetes and makes it easy to deploy various services within existing or newly created Kubernetes clusters. Before using the Helm chart for installation, users need to ensure that they have downloaded and configured Helm to work with their Kubernetes cluster. From there, it is a three-step process to install Consul. First, clone the HashiCorp Consul Helm chart onto your machine. From there, verify that you have the most recent version by running the helm inspect command. Consul is now ready to be installed with the default configurations, but it is possible to customize your Consul deployment by creating a values.yaml file. For the full list of possible configuration values, please visit our Consul docs. Once you have determined your preferred configuration of Consul, run the following command:

$ helm install --name consul -f ./values.yaml ./consul-helm

As stated above, this example indicates a custom configuration with the values.yaml and -f flag added. To run the install with the defaults, remove these values from your command. To learn more about the Consul Helm chart and what it supports, read this blog. Once the configuration is completed, you’ll have installed Consul onto your Kubernetes cluster.

Another tool to help get you started using Consul and Vault on Kubernetes is Shipyard. Shipyard was created by Nic Jackson, developer advocate at HashiCorp, and is able to deploy local Kubernetes clusters with Consul and Vault installed using a single command. The guide includes instructions for deploying, interacting with, and destroying new Kubernetes clusters run locally with Docker. More information on how to develop custom cluster configurations and exposing ports will be added to the docs in the future.

Lastly, if you are looking for additional help for getting started running Consul on Kubernetes, we offer a series of guides on our Learn page. These guides range from introductory topics, like basic installation, to more complex tracks, like production ready deployments. With Consul now installed on Kubernetes, the next step is to turn to the robust ecosystem of Consul integrations to help optimize your Kubernetes deployments.

»Service Discovery and Network Middleware Automation

Before diving into Consul’s service mesh capabilities, there is an important consideration that organizations need to keep in mind: service discovery. Kubernetes offers the ability to run an incredibly large number of services and offers its own service discovery capabilities, but for the most part, enterprises are migrating to Kubernetes while still maintaining their existing application environments. This means that services may be spread across multiple datacenters and tracked by multiple registries. Deploying a solution to centralize service discovery across all runtime platforms will ultimately make it easier to manage these deployments and reduce some potential friction in the transition to containerization. Consul can register services through either a configuration file, added at the time of deployment, or dynamically through the HTTP API. Once services are registered, Consul also provides the ability to run health checks to reduce the risk of outages. Through integrations with popular network middleware solutions like F5 BIG-IP and HAProxy, service discovery and health checks can be done across multiple environments.

The way Consul interacts with network middleware is based on the idea of managing east-west communication (internal service to service) versus north-south (external users to internal services). Load balancers are a trusted way to securely control the flow of communication into an organization’s networks, but the need for static IPs may slow down service deployments and doesn’t scale in a way we might need to fully migrate to microservices. To alleviate this challenge, Consul can dynamically provide service data and health status back to a load balancer, making it visible to the operators that are managing the network health. This enables organizations to still utilize the load balancing solutions they currently trust, without creating a bottleneck for service deployment and reduces the chance of unknown services operating within a given environment. For a deeper dive into how these integrations work, read this blog for F5 BIG-IP or try out this Learn track for HAProxy.

»Envoy Proxy and Ambassador API Gateway

As mentioned earlier, Consul helps manage east-west communication within a given environment. This is done by using proxies for service-to-service communication inside Kubernetes environments. Proxies are very beneficial to managing microservices at scale by reducing latency in service calls, enabling greater security using mTLS, and helping monitor system resources for the services they are attached to. Consul offers a built-in L4 communication proxy for testing and development, but for proxying L4 and L7 communication in production, Consul service mesh leverages Envoy. Through the integration with Envoy, Consul service mesh users can automate a number of features that are critical for managing microservice deployments at scale including service discovery, health checks, and load balancing. Since Envoy is supported as a first class integration, Consul can automatically generate dynamic configurations for new proxies, upstream options, and mesh gateways. To try using Envoy with Consul, follow this Learn guide. When configured properly, a service mesh can be extremely helpful for organizations that are moving towards containerized applications, but the end goal is to eventually expose those applications to users that reside outside the network. For that, we recently launched an integration with Datawire’s Ambassador API gateway.

Datawire Ambassador is Kubernetes-native, ingress controller designed to help route communication and service requests from users and applications outside of a Kubernetes cluster. Ingress controllers are a set of rules that govern how external requests will be routed within the cluster and are critical to securing end users access. Beyond just routing rules though, Ambassador also supports multiple request types (HTTP, gRPC, etc.), service testing, authentication, and security. Ambassador is built on Envoy, which means that it can communicate with proxies created and managed by Consul. For example, let’s say a new service has been deployed within a cluster that is using Consul service mesh. If configured properly, Consul will register that service, attach a sidecar proxy to it, and communicate its existence with the Ambassador gateway. When a user sends a request for that service, Ambassador and Consul will facilitate the connection via Envoy and encrypt it using TLS. For more information about the integration with Ambassador, read our documentation.

»Distributed Tracing with APM Solutions

Consul’s integrations can not only help with service discovery and managing service-to-service communication, but also with monitoring and observability. HashiCorp Consul has the ability to export telemetry data to Datadog, AppDynamics, and SignalFX and enable distributed tracing. Consul uses either statsite or statsd to forward telemetry data, such as network health, which is then imported into these platforms for use in custom dashboards. Distributed tracing goes beyond this to include specific service request information that can be extremely helpful for managing microservices. This blog describes some of these capabilities and contains some demos to help users get started implementing tracing in their own environment.

»The Consul Ecosystem

The Consul ecosystem is continuing to grow to meet the demands of our customers. While we’ve highlighted a number of integrations in this blog, there are many others that we didn’t touch on that are also currently available. Our goal is to ensure that Consul is able to provide organizations with a single control plane for managing any runtime platform, like Kubernetes. To learn more about our growing ecosystem, be sure to stop by our booth at KubeCon and speak with our experts. If you are part of an organization that is interested in building a Consul integration, please read this page to learn more about the Consul Integration Program.