Consul Service Mesh Support for AWS Lambda Now Generally Available

We’re pleased to announce the general availability of HashiCorp Consul support for mesh services invoking AWS Lambda functions. In addition, we’ve released a public beta for AWS Lambda functions invoking mesh services. This release means service mesh users can now have consistent workflows for encrypted communications flowing between mesh services and Lambda functions.

Lambda support is enabled in the newly released Consul 1.14. In this blog post, we’ll examine the intersection of serverless functions and service mesh from a broad perspective and then show you how this integration works in practice with some code examples.

»Serverless Functions

Commercial functions as a service runtimes like AWS Lambda offer serverless functions that enable users to run code in response to specific events. In this event-driven model, applications are broken up into modular functions that can be easily invoked and scaled up and down. Serverless functions eliminate the need for developers to manage the backend infrastructure and configure the server software needed to run application code.

Because all of this management is handled by the cloud provider, developers can focus on writing the business logic for their applications. According to Datadog's 2022 The State of Serverless report, more than 50% of companies operating in the cloud are using serverless technologies, with Lambda being the frontrunner in popularity.

The advantages of serverless functions include:

• Costs: You pay only for the compute resources you use.
• High availability: The platform provides redundant code deployment and autoscaling.
• Productivity: The cloud provider is responsible for provisioning and managing the compute infrastructure so your team doesn’t have to.

Despite these advantages, there are challenges that come with onboarding serverless workloads into production environments. This is especially true when dealing with distributed systems deployed across a variety of platforms, providers, and runtimes:

• Service discovery: Services need to be able to discover and locate serverless functions.
• Secure communication: Traffic between services needs to be verified and encrypted.
• Consistent operational workflows: Serverless functions need to be incorporated into the same operational framework as other workloads, without the need to include exceptions, deviations, and workarounds.

»Consul Support for AWS Lambda

HashiCorp Consul is an enterprise-ready networking solution that provides a service mesh built to address these challenges. Consul service mesh:

• Allows you to connect all your workloads and take advantage of features like identity-based security, traffic management, encrypted communication, and operational insights
• Can be deployed to hybrid environments, including on-premises and multi-cloud infrastructure
• Supports multiple runtimes, including virtual machines (VMs), Kubernetes, Amazon Elastic Container Service (Amazon ECS), and HashiCorp Nomad

Consul now has first-class runtime support for AWS Lambda. AWS Lambda functions can be fully integrated with your service mesh and benefit from the same reliability, observability, and security features that Consul provides. For example, you can use Consul intentions and traffic management on Lambda functions the same way you would use these features with other services.

»Integrating AWS Lambda Functions

Registering Lambda functions into Consul is automated using the Consul Lambda service registrator. This integration provides a simple way to register AWS Lambda functions into the service mesh.

First, you use the Consul Lambda registrator Terraform module to deploy Lambda registrator. Lambda registrator automatically registers all Lambda functions that you have tagged with serverless.consul.hashicorp.com/v1alpha1/lambda/enabled: true into the service mesh. This happens in four steps:

1. Whenever you create, delete, tag, or untag Lambda functions, AWS logs the events into CloudTrail.
2. EventBridge routes the events to the Lambda registrator.
3. Lambda registrator registers the Lambda function as a Consul service and stores a service-defaults configuration entry.
4. Lambda registrator retrieves the information needed to make mTLS connections into the mesh from Consul and stores this data, encrypted, in AWS Systems Manager Parameter Store.

»Consul Mesh Services Invoking AWS Lambda Functions (GA)

Applications that call AWS Lambda functions can do so through a terminating gateway or directly through their Envoy sidecar proxy:

Once registered, you need to set the Lambda service as an upstream for any services that will use it. Then you can call it like any other mesh service:

»AWS Lambda Functions Accessing Consul Mesh Services (Beta)

Lambda functions that call into the mesh use the new Consul Lambda extension to make mTLS requests to services through a mesh gateway. The extension runs within the execution environment of your Lambda function and performs three main steps:

1. The extension retrieves the mTLS data that Lambda registrator stores in AWS Systems Manager Parameter Store from the path configured by CONSUL_EXTENSION_DATA_PREFIX. The extension periodically updates the mTLS data to ensure that it stays in sync with Consul.
2. The extension creates a lightweight TCP proxy that listens on localhost for each upstream port defined in CONSUL_SERVICE_UPSTREAMS.
3. For each request received from the Lambda function on its upstream listeners, the extension establishes an mTLS connection with, and proxies the request through the mesh gateway configured by CONSUL_MESH_GATEWAY_URI:

To use the Consul Lambda extension, you add it to your Lambda function as a layer. Then you set environment variables on your function to configure it. Your function doesn’t need to know anything about the extension, so there should be no impact to your existing code.

You can download the Consul Lambda extension from the HashiCorp releases website at the Consul Lambda releases page.

The following Terraform configuration creates a Lambda layer from the consul-lambda-extension zip file and a Lambda function that uses that layer:

To call the example-service upstream, the function makes requests to localhost:1234.

»Next Steps for HashiCorp Consul

Our goal with HashiCorp Consul is to provide a consistent, enterprise-ready control plane to discover and securely connect any application. Users can now further broaden their service mesh implementations by utilizing these new Consul improvements for AWS Lambda functions.

To learn more about how to use Consul and Lambda together, please visit the Consul and AWS Lambda documentation.