Application Feature Toggles with HashiCorp Consul

Feature toggles (also known as feature flags) conditionally deliver application features to users. We previously discussed feature toggling of HashiCorp Terraform resources to mitigate the impact of production infrastructure changes. In this post, we demonstrate how to manage application feature toggles when using HashiCorp Consul and consul-template.

In applications, we use different types of feature toggles to quickly change functionality of application code. Not only can we use release toggles to hide a release but also experimental toggles to initiate experiments for A/B testing and operational toggles for rolling back problematic changes. Using Consul, we can track, manage, and update our various toggles in a single location and retrieve updated values at runtime instead of restarting the application for changes to flags.

»Consul Setup for Feature Toggles

If you want to self-manage your feature toggles or already use Consul’s service discovery or service mesh features, you can configure Consul KV (key-value store) to store your flags or toggle based on the traffic routing configuration for the upstream service. First, you deploy a Consul server and client. When your application accesses the client to retrieve values from Consul KV, the client forwards the request to the server. A client should be deployed on the same node as your workload. For example, if you deploy Consul in a container orchestrator such as Nomad or Kubernetes, you can reference the Consul client deployed to the local node. In a virtual machine environment, you can deploy a Consul client for each virtual machine that hosts an application.

To demonstrate feature toggling with Consul KV, let’s use an ASP.NET Core service called the report service that returns a JSON payload with a report totaling and listing travel expenses for a given trip.

{
    "tripId": "d7fd4bf6-aeb9-45a0-b671-85dfc4d09544",
    "expenses": [
        // omitted for clarity
    ],
    "total": 25.5700
}

»Toggle with Consul KV

As an example, let’s say your users ask you to add a feature showing the number of expenses for the trip. As your team develops the feature, you do not want to release its functionality while committing and pushing other changes to production. Therefore, you need to hide the feature behind a transient release toggle. You update the report creation logic to add the number of expenses when you enable the feature toggle.

   private void addNumItems(ReportTotal reportTotal)
   {
     if (_toggleClient.GetToggleValue("enable-number-of-items").Result)
     {
       reportTotal.NumberOfExpenses = reportTotal.Expenses.Count;
     }
   }

The _toggleClient abstraction calls a function called GetToggleValue in order to retrieve the feature decision from Consul KV.

   public async Task<bool> GetToggleValue(string name)
   {
     var getPair = await _consulClient.KV.Get("toggles/" + name);
     if (getPair.StatusCode != System.Net.HttpStatusCode.OK)
     {
       return false;
     }
     var value = Encoding.UTF8.GetString(getPair.Response.Value, 0, getPair.Response.Value.Length);
     return TransformToggleValueToBoolean(value);
   }

GetToggleValue uses the Consul.NET library to retrieve the value for a given key in Consul with the Consul API. The Consul documentation includes a list of libraries and SDKs for a number of programming languages to interface with the Consul API. This example always disables the toggle if the application cannot access Consul or the toggle does not yet exist in the key-value store at the /toggles path. You can customize this path to isolate the feature toggle based on different toggle categories, application services, or development teams. In addition to customizing the path for your feature toggle, you can include additional toggle metadata such as timestamp, ownership, and category. In the example, the toggle exists as a binary true or false.

The code automatically ignores the feature to output the number of expenses when the toggle does not exist. Users continue to see the JSON payload with a list of expenses and total. When your team progresses with feature implementation, you use the Consul command line or dashboard to create a disabled feature toggle for the number of expense items.

$ consul kv put toggles/enable-number-of-items false

To release the feature, you enable the toggle through another command or dashboard update by setting it to true.

$ consul kv put toggles/enable-number-of-items true

Now that you enabled the feature, your user should receive a payload with the number of expenses. You can confirm this by making a request to your report service and receive a report with the number of expense items for the trip.

{
    "tripId": "d7fd4bf6-aeb9-45a0-b671-85dfc4d09544",
    "expenses": [
        // omitted for clarity
    ],
    "total": 25.5700,
    "numberOfExpenses": 200
}

After releasing the feature, you allow the toggle to remain enabled for a few weeks to allow for regression testing. Once you confirm that the new release did not break existing functionality, you remove the code for the toggle.

»Scaling Toggles for Production

When your system uses many toggles, you might need to audit the lifetime of toggles to prevent technical debt and control access to toggles to prevent unapproved releases. With Consul KV, you can create Consul watches to monitor modifications to your toggle key prefix. In order to control access to specific toggles, you can use access control lists (ACLs) to prevent unauthorized changes to flags and namespaces to further isolate access by teams. For additional ease of management, you can use feature flag management tools such as LaunchDarkly to manage toggles.

For a few feature toggles, you can write your toggle logic to make an HTTP request to Consul’s key-value store each time the code path executes. However, as you add more flags to your application and scale your feature toggling approach, you will need to ensure that the code proceeds with the correct logic if the application cannot access the key-value store due to network issues. For example, when using LaunchDarkly to manage feature toggles, you can cache flag information to Consul with the persistent feature store if the application cannot reach LaunchDarkly.

When using Consul for toggling at production scale, you will need to cache the responses from Consul KV for feature decisioning and update the flag values in the background. As a result, your application can reference the cached values even if it cannot reach Consul. In the example, you could refactor GetToggleValue with a background task to asynchronously read values from Consul. However, this approach requires additional logic and testing for edge cases in Consul’s behavior, such as reverse indexing on state reloads. Instead of writing and testing logic within your own library, you can use consul-template to write the values to file. consul-template populates values from Consul into the filesystem and handles its edge cases.

In the example, you can deploy consul-template on the node that hosts the report service. You can configure consul-template with a template configuration to read all of the toggles under toggles in Consul KV and write them to a file.

template {
 contents = <<EOH
       {{ range ls "toggles" }}
       {{ .Key }}={{ .Value }}{{ end }}
       EOH
 
 destination = "/etc/report/toggles"
 
 wait {
   min = "2s"
   max = "5s"
 }
}

The template configuration reads the keys and values under the toggles path in Consul KV and edits the /etc/report/toggles file to reflect any changes. For example, when you add the enable-number-of-items feature toggle, you can see the /etc/report/toggles file update with the current value.

enable-number-of-items=false

When using consul-template and rendering to a file, you can update the GetToggleValue function to read the toggle key and value from the file instead of making an HTTP request to Consul.

   public async Task<bool> GetToggleValue(string name)
   {
     var lines = await ReadLines();
     foreach (string line in lines) {
       if (line.Contains(name)) {
         return TransformToggleValueToBoolean(line);
       }
     }
     return false;
   }

Meanwhile, consul-template updates the file automatically when it detects changes to Consul KV.

»Conclusion

We examined how to use Consul KV to store feature toggles and release a feature in an example application. In addition, we discussed feature toggle management in the form of Consul watches and ACLs. For a production scale, we examined how to refactor our code to cache and update toggles with consul-template. To learn more about Consul, see the HashiCorp Learn guides. For more configuration parameters and additional uses for consul-template, check out its repository.

Post questions about this blog to the community forum!