A brief history on how to secure your workflow for job deployments in Nomad

Nicolas Corrarello is a Regional Director for Solutions Engineering at HashiCorp based out of London.

A lot of HashiCorp users and employees love our whole suite of products, but like with your grandmother, it’s almost impossible not to have one slight predilection over one of our creations (says the favorite grandchild).

In my case, my predilection for HashiCorp Nomad is quite evident, to the point of being a running joke among the European team. I think not a single customer meeting goes by without someone saying "yeah, that’s Nomad, Nico’s favorite product…".

As such, you can imagine my excitement, when in September last year, Armon took the stage to publicly announce the plethora of features we were shipping in Nomad 0.7. I finally had the completeness of vision in regards to an enterprise scheduler. Quotas and Namespaces minimised the already low operational overhead for our enterprise customers, and ACLs would provide secure access to our open source users. On that last point, and exactly at the time I took that picture I had a short epiphany (and not many people have actual documentation of those). How would a team operationalise ACLs?

• An operator could create long lived tokens for users and systems doing deployments (like CI Pipelines). Definitely not.
• Shared tokens? Don’t get me started.
• Build LDAP Integration for users and long lived tokens for machines. Sounds like a lot of work.

If only we had a technology that could broker credential access, and had already a way to establish identity using something like LDAP (for users), and something like, I don’t know, EC2 for CI workers (which can technically run in Nomad as well). Wait… we actually have something. HashiCorp Vault was actually the perfect fit. I was already playing the workflow in my head:

1. A user authenticates against Vault using LDAP, gets a Vault Token
2. That Vault token is tied to a specific Vault policy, which in turn is tied to a path in Vault that can generate the a Nomad token with the exact Nomad policy.
3. User requests a Nomad Token with a short expiration time
4. Interacts with Nomad as desired
5. Happiness :D

Only one small issue, we didn’t have (at the time), a Nomad Secret Engine. Now something you definitely need to know about this company, is that a good idea is never enough, you need to execute on it. So with limited internal knowledge of Vault, and having never written a single line of Go, there I went.

The result is actually a testament to how amazing Vault is, thing appeared to just work as I were coding them, and about three hours later, Vault was actually generating and revoking Nomad tokens. A long code review later (https://github.com/hashicorp/vault/pull/3401) the secret engine was merged in 0.9.1. So let’s have a quick look and how to quickly get it set up.

ACLs of course need to be enabled in Nomad, through the acl stanza in the configuration file:

acl {
  enabled = true
  token_ttl = "30s"
  policy_ttl = "60s"
}

The cluster should be bootstrapped for ACLs, and you should have an initial management token. It’s worth noting that the initial management token can be revoked, but it may alter child tokens, so you most likely don’t want to give that token to Vault, but rather generate a child one that can be managed independently:

$ NOMAD_TOKEN=${INITIAL MANAGEMENT TOKEN} nomad acl token create -type=management
Accessor ID  = 5f7d8875-1bf9-ed80-c8c3-e4e6e20c2408
Secret ID    = ea83681b-9ca8-4393-999f-a25731ad3b55
Name         = <none>
Type         = management
Global       = false
Policies     = n/a
Create Time  = 2018-03-17 11:36:12.696245 +0000 UTC
Create Index = 8
Modify Index = 8

You should also have a set of policies created in Nomad, as Vault will leverage those policies to ultimately create tokens, in this case, you can create a sample policy that looks like this:

namespace "default" {
    policy = "read"
}
agent {
    policy = "read"
}
node {
    policy = "read"
}

And import it into Nomad:

$ NOMAD_TOKEN=${INITIAL MANAGEMENT TOKEN} nomad acl policy apply policyone ./payload.json
Successfully wrote "policyone" ACL policy!

Off to configure Vault now. Start by mounting the nomad secret engine into a mountpoint and configure the connection to Nomad:

$ vault mount nomad
Successfully mounted 'nomad' at 'nomad'!
$ vault write nomad/config/access \
    address=http://127.0.0.1:4646 \
    token=adf4238a-882b-9ddc-4a9d-5b6758e4159e
Success! Data written to: nomad/config/access

Now create a role within Vault to issue Nomad tokens associated with the Nomad policy we have just created:

$ vault write nomad/role/role-name policy=policyone
Success! Data written to: nomad/roles/role-name

And a Vault policy to generate Vault tokens that allow the creation of Nomad tokens in that role:

$ echo 'path "nomad/creds/role-name" {
  capabilities = ["read"]
}' | vault policy-write nomad-user-policy -
Policy 'nomad-user-policy' written.

As a last step, for example, you can just generate a Vault token associated with the policy, and retrieve a Nomad token as shown below:

$ vault token-create -policy=nomad-user-policy
Key             Value
---             -----
token           deedfa83-99b5-34a1-278d-e8fb76809a5b
token_accessor  fd185371-7d80-8011-4f45-1bb3af2c2733
token_duration  768h0m0s
token_renewable true
token_policies  [default nomad-user-policy]
$ vault read nomad/creds/role-name
Key             Value
---             -----
lease_id        nomad/creds/test/6fb22e25-0cd1-b4c9-494e-aba330c317b9
lease_duration  768h0m0s
lease_renewable true
accessor_id     10b8fb49-7024-2126-8683-ab355b581db2
secret_id       8898d19c-e5b3-35e4-649e-4153d63fbea9

This technique can be combined to do zero touch provisioning with anything that authenticates with Vault, and ultimately provides a very secure workflow for Nomad management and scheduling.

Are you interested in telling others your HashiCorp story or perhaps how HashiCorp products helped with that amazing thing you built? Let us know. Email your story or idea to guestblogs@hashicorp.com