HashiCorp Nomad Enterprise Audit Logging

Nomad 0.11 Enterprise includes a new way to allow administrators to gain insights into what user actions are being made on a Nomad cluster and fulfill necessary compliance requirements. With enterprise audit logging, operators will be able to answer questions around what is happening on Nomad cluster, who is invoking the action, when it happened and if the request was successful or not.

In this blog post, we’ll go over how audit logging is implemented in Nomad, how the audit log is constructed, and what control administrators have over what is audited.

»Audit Logging Overview

Nomad Enterprise audits at the HTTP API level. An overview of Nomad’s HTTP API can be found here. Since server and client nodes both serve the HTTP API, both servers and clients should be configured to use auditing.

In order to ensure that a request is properly audited throughout its lifecycle, a single request contains multiple audit log entries. These entries correspond to a particular stage.Currently there are two stages, OperationReceivedand OperationComplete.

The picture above shows the request flow for an HTTP request with audit logging enabled. The OperationReceived stage ensures that Nomad is able to audit the request before we potentially modify the state of the cluster. The OperationComplete stage contains the same information as the OperationReceived event with additional information relating to the Response returned by the invoked handler (status code and any error messages).

Guaranteeing delivery of audit logs is important from a compliance perspective. Nomad requires you to choose one of two delivery guarantee modes (enforced and best-effort) which are important to understand when Nomad encounters an error in the auditing pipeline (such as the audit log file running out of disk space).

• enforced - With delivery_guarantee set to enforced, an audit event that is not filtered, must be successfully written to its sink in order for the request to be invoked by the handler. If the failure occurs during OperationComplete, after the handler was invoked, the response from the handler will be replaced with the error from the audit log.
• best-effort - With delivery_guarantee set to best-effort, an audit event that fails to successfully be written to its sink will be logged as an error, and not block the request from being invoked.

These two modes allow operators to choose their level of fault tolerance vs. risk acceptability. A Delivery Guarantee of enforced is recommended as it is the only way to ensure requests are fully audited.

»Configuring the Audit Logger

Audit Logging in Nomad Enterprise can be configured with minimal setup, but also allows for fine-grained controls and file rotation, as well as configurable filters to prevent certain operations from being audit logged.

A base audit log configuration is as simple as:

audit {
  enabled = true
}

This will configure audit logging with maximum security and verbosity. This configuration doesn’t filter out any events and creates an audit log file in your specified data_dir directory (at the path $data_dir/audit/audit.log). This configuration also configures the sink to have a delivery_guarantee of “enforced”.

If we want more control over the audit file we can configure it:

audit {
  enabled = true

  sink "audit file" {
    type               = "file"
    delivery_guarantee = "enforced"
    format             = "json"
    path               = "/var/lib/nomad/audit/audit.log"
    rotate_duration    = "24h"
    rotate_max_files   = 10
  }
}

This example configures the audit log for daily rotation and to only keep the latest 10 rotated files.

What if we want to limit the amount of logging we do? Although logging all stages and all events is the most secure option, we may want to only audit log the OperationComplete stage or completely ignore certain endpoints from our logs. We can use filter stanzas to filter out matching events from being audit logged. Below is an example configuration with a filter:

audit {
  enabled = true

  filter "operation received events" {
    type = "HTTPEvent"
    endpoints = ["*"]
    operations = ["*"]
    stages = ["OperationReceived"] 
  }
}

This filter will apply to HTTPEvents and to all endpoints and operations (HTTP verbs in this case) for the OperationReceived stage. This means that any matching endpoint + operation (in this case all endpoints and operations) will be filtered out and excluded from the audit log for the OperationReceived stage. They will still be audit logged for the OperationComplete stage.

»Audit Log Entries

The example below shows two audit log entries for a request made to a Nomad server using the CLI with the following command: nomad node status . We can see the entries for the two stages associated with this request, along with information from the auth token that was used to create the request as well as some request metadata. The OperationComplete event contains the contents from the OperationReceived entry, along with an added response key.

{
  "created_at": "2020-03-24T13:09:35.703869927-04:00",
  "event_type": "audit",
  "payload": {
    "id": "8b826146-b264-af15-6526-29cb905145aa",
    "stage": "OperationReceived",
    "type": "audit",
    "timestamp": "2020-03-24T13:09:35.703865005-04:00",
    "version": 1,
    "auth": {
      "accessor_id": "a162f017-bcf7-900c-e22a-a2a8cbbcef53",
      "name": "Bootstrap Token",
      "global": true,
      "create_time": "2020-03-24T17:08:35.086591881Z"
    },
    "request": {
      "id": "02f0ac35-c7e8-0871-5a58-ee9dbc0a70ea",
      "operation": "GET",
      "endpoint": "/v1/nodes",
      "namespace": {
        "id": "default"
      },
      "request_meta": {
        "remote_address": "127.0.0.1:33648",
        "user_agent": "Go-http-client/1.1"
      },
      "node_meta": {
        "ip": "127.0.0.1:4646"
      }
    }
  }
}
{
  "created_at": "2020-03-24T13:09:35.704224536-04:00",
  "event_type": "audit",
  "payload": {
    "id": "8b826146-b264-af15-6526-29cb905145aa",
    "stage": "OperationComplete",
    "type": "audit",
    "timestamp": "2020-03-24T13:09:35.703865005-04:00",
    "version": 1,
    "auth": {
      "accessor_id": "a162f017-bcf7-900c-e22a-a2a8cbbcef53",
      "name": "Bootstrap Token",
      "global": true,
      "create_time": "2020-03-24T17:08:35.086591881Z"
    },
    "request": {
      "id": "02f0ac35-c7e8-0871-5a58-ee9dbc0a70ea",
      "operation": "GET",
      "endpoint": "/v1/nodes",
      "namespace": {
        "id": "default"
      },
      "request_meta": {
        "remote_address": "127.0.0.1:33648",
        "user_agent": "Go-http-client/1.1"
      },
      "node_meta": {
        "ip": "127.0.0.1:4646"
      }
    },
    "response": {
      "status_code": 200
    }
  }
}

»Getting Started

We're releasing Nomad's Enterprise Audit Logging feature as part of the Governance & Policy Enterprise Module in beta for Nomad 0.11 to get feedback from our practitioners. We’d like to hear what other auditing, eventing, and observability features you are interested in seeing. If you would like to try it out, please reach out to your Technical Account Manager to request a download link and give us feedback in the issue tracker.