Migrating From Kubernetes to Nomad & Consul and Doubling Pipeline Speed With GitLab at Internet Archive

Learn how archive.org (The Internet Archive) moved workloads from Kubernetes to Nomad and Consul in minutes using GitLab with a project.hcl template.

Speaker: Tracey Jaquith

Slides: https://archive.org/~tracey/slides/hashitalks/#/

»Transcript

Welcome to HashiConf Europe. Thanks for attending my talk. My name is Tracey Jaquith, and this talk is GitLab + Nomad = A Dream Come True. I have a minor in math, and that is math.

My handle is @tracey_pooh on Twitter. I am from the Internet Archive. I've been there for over 20 years now. I'm the second-longest person there, aside from our founder. My title is TV Architect, and I work on video, audio, DevOps, and JavaScript.

The slides are at tracey.dev.archive.org if you want to follow along. We're going to be using some code and techniques from a repo linked below at gitlab.com/internetarchive/nomad.

You probably don't know me, but whether you're my friend, or family, or colleague, or whatever, at some point, our discussion will start going towards containers — whether you like it or not. That's something to know. If you invite me in, you get a Trojan horse. I love containers.

Why are we here? And I don't mean it in a French existential way, although that is interesting. We are here at this talk to see how easy it is to use Gitlab with Nomad clusters. GitLab already works with Kubernetes out of the can — we want to make it work with Nomad just as easy. In fact, better. Here's GitLab + Nomad, we're going to do it with a two-line YAML file and two CI/CD variables, and that's it — and we'll have a demo to show it too. 

»Approach and Goals

We want to replace Kubernetes with Nomad. We want to use GitLab's normal auto DevOps — they call it pipelines — so we can do their build phase, which works great. It'll push to the registry, then we can use the test — all that works great, no need to change that. We just have to customize the deploy phase — straightforward. And again, last reminder, tracy.dev.archive.org, to follow along.

Here's how we're going to do it with stick figures. Your devs are going to commit and push like normal. That will kick off a CI/CD normal pipeline with Gitlab. Their standard pipeline is great. It will start off with a build phase. It'll then move to a test phase — that's optional —‚ but you should be testing. Then it will move to a deploy phase, which is what we’ll customize.

»How We’ll Do It 

We've got a slew of variables from GitLab. They all start with CI so we're going to make use of those and leverage those. We also have these environment variables that we’ll use for customization called Nomad_VAR and Nomad_secret. 

Here's a couple of examples. We've got a commit_sha here — that's the current commit. The checksum is the thing running through the pipeline — or the latest thing from Git. And we've got this nomad_var_slug, which is a way to customize and make nice hostnames and semantic goodies and goodness. We're going to combine all those into a project.nomad generic template. Then that will ship off to Nomad with a nomad.run — right to your Nomad cluster. That's our deploy phase.

It's a few minutes in. You might be feeling a little tired. Maybe you've been watching a lot of talks, but here's the too-long-didn't-read. All you need is two settings, two CI/CD variables. You put that in your GitLab repository — I like to put it in the group, one level higher — and that'll apply to all your projects. 

You add your nomad_token for whatever your cluster is — that's a fake token. Then you do a two-line YAML file include. That's all you need to include. You would put this file as a GitLab.ci.yml at the top of each of your repos, and you never have to change it because everything's getting included.

We are including another GitLab.ci.yml file — so what's in that included GitLab.ci.yml file? Well, there's some other stuff, but it's focusing on the deploy. You start the deploy. You start a little script off to figure out how to pull the deploy off. You pull down this project Nomad file. Nice and easy. We're going to rename it to call it HCL. Then we switch out this Nomad VAR slug inline —that's because we can't quite do that in the project HCL right now with the HCL spec. We don't have to worry about that too much. 

We take all the CI variables, and we're going to stuff it into a little JSON-like encoded environment variable. We can use it in the project template. We do the same thing with a nomad_secret in a slightly different version, but it's a slightly different format. —Then we're going to pull those two environment files in with the project HCL, combine those, and send those two files to Nomad with a Nomad run. And it's off to Nomad with your deploy jobs — it's deploying. If you want to see more and see all of this, you can look at the link below that says full GitLab-ci.yml file.

»Nomad Project

That's including a nomad-project, which we talked about combining the variables. It's a generic template linked above, and it will use your GitLab CI/CD variables. Here's a couple of other examples. Your ref slug is the branch name of the pipeline that's running. And the CI registry image is the current docker registry image that's been pushed to Docker from your Docker build and test phase — we're going to use that.

You might have some customizations — maybe you've got a memory or CPU tweak. You can do that easily. You might have a secret, like a database URL. This will all leverage HCLv2 — which is great, thank you so much for it — which allows you to do things like loop over things in your config. In this case, we're going to loop over a bunch of ports that will be public out to your web servers. Maybe we've got a Postgres port that also needs to be accessible within your job task — we can loop over those. 

This little thing here, all that's doing is making it easy — so we can refer to things by a port name rather than port number. You can see more examples by clicking on some of those links.

»Minimal Project

Let's say you want to try a minimal project. You don't want to get like the whole thing — you want to try the minimal project you can use with GitLab. We're going to show you how to do that right here. We're going to call the job, hello world. That's how you start your Nomad job stack. We're going to start with another group. That's going to group a few things. we're going to make a web server, and it will be running on port 5000.

We're going to set up a task as part of the group. That's how you do this. And we're going to say, we want you to run Docker. That Docker image is using that CI registry image — talking back to your registry; the thing that was just built — it's a little bit of a longer thing — but it's using those CI/CD variables that come right from GitLab. It's saying I am going to be running on port 5000 or as HTTP.

Most people's GitLabs are a private registry — so, we're going to set it up so you can do the Docker logins. That's this little auth stanza. That again comes right from GitLab — Use that easy-peasy. Then we do a little service stanza. The service stanza will tell the load balancer I want you to use this new hostname.

Nomad VAR Slugs 

We're going to use these VAR slugs — that'll give you a nice semantic hostname. We're going to be using it on port HTTP. And — while we're at it in the service descriptor — you can set up a little health check task. 

Consul and Fabio — the load balancer — will see that and will check every ten seconds with latency of up to two seconds — and will want to see the slash page with a 200 request.

If you have Nomad, you could try this out right now. You've got your Nomad ADDR, Nomad token in your environment. You can pull down this file and run it — it should just work.

Talking about the Nomad VAR slugs. It's nice to have hostnames that don't look like a Bitly address. Bitly is nice, but it’s nice to have nice hostnames. We use the GitLab group and the GitLab project — and they're slugified and dashed in between. Those come right from GitLab. If you're not using the main or master branch, we'll add that the branch name —‚ slugified on the end too —so you know exactly what you're testing or looking at.

»Nomad Job Names

The Nomad job names also get slugified in that same way — they have the same semantic slug. Right here on the left, we've got the job name, and we can see that — for example — the Internet Archive Group is running two different projects in this Nomad cluster. One's called word salad — a fun game. The other one is WWW, which has two branches —the main branch and some feature branch that someone's working on. Everything's happy. We're all happy.

»Demo

Now it's time for demo time. Yay, demo time. We're going to create a new GitLab repo and deploy it to Nomad. We're going to do it as a hello world classic thing in three files and in slightly less than ten lines.

Here we are with a clock JS file, we're going to do a simple Node.js. That does the minimal HTTP server. It will answer the time and date for every request that comes in. It listens on our port 5000 — our main web port.

Then we add a Docker file, which is pulling from node:alpine — the fast version — which is optimized for node NPM, and yarn. We're going to copy all the files into /app. Then —when the container runs with that last line — we’re going to ask the node to run file app.clock.js. Then we just include our two-line GitLab CI.yml file. Nice and easy-peasy.

Here's our demo. And again, just three files — make it nice and easy. I'm going to copy the first few lines for the clock.js. I'm going to switch to an open tab where I've got a group already set up in Internet Archive — going to make a new project from scratch.

I'm going to create a blank project. We'll call it hashiconf-demo — nice and simple. Then we'll do a read me — that's fine; otherwise, we'll create a new blank project. 

Now, to make things easier — because you can do this with GitLab GUI — you can add files right from the browser. We're going to open three tabs for our three files. It will make it a little bit easier and quicker for them to load. And we're going to start our first file, drop in our little JavaScript fragment —we're going to call it clock.js. And now code highlights, which is kind of nice. And again, it will listen on 5000 and answer the time for every request that comes in. I'll commit the changes. This will add the file to the repo. Nice and easy. Then, we can get rid of this tab and go for the next file. 

We're going to grab our little Docker file — a three-line file — and start it out. Again, it's a real simple node app —so, it should run and build super-fast. We don't need any extra dependencies because that little web server is already built in. 

We should be good to go. We'll do the same thing — commit the file, send it off to the repo. Now we copy the GitLab ci.yml file — that's our little magic two-liner that's pulling from the Internet Archive Project. We make a little type — we don't need a template — we're going to make one and commit the changes.

Now we've got three files — GitLab ci.yml file, and the Docker file are enough to start a pipeline. So, we should already see some pipelines firing off from GitLab. There they are. One has already been auto-canceled because they're redundant— and that's fine. 

Let's see how the build is doing. Here's the build — and that's super quick. That node image is super small, and we're doing almost nothing — it's already built, very quickly.

Now let's go back to the pipeline and see how the next phase is doing. Let's start a custom deploy phase and click on that. It will be pulling that registry image. It'll lead — sending off to Nomad — and there's our nice semantic name. It shows you what name is deploying to — that's clickable. 

If you look up above at GitLab.com, it's just like the project and group. We click on it — and there we go; that fast — super-easy. It shows us the date and time, easy peasy — just a few minutes. You can be a web app person too. By the way, this is both responsive — I'm kidding. It is a responsive design, though.

How did that all go down? So normally, you're talking about people talking to a browser, that's talking to HTTPS to your web app —or the thing that's running that's usually going to talk to a load balancer or your load balancers — probably going to terminate the traffic to HTTP. Then, send that HTTP traffic onto your little web app container or whatever it is that's running. Often, it's a web daemon of some sort — Nginx, or otherwise. It might have a database or some Redis or something behind that can all be grouped in there — mentally speaking. 

But when you're using Nomad and GitLab, you're going to be using these GitLab CI/CD pipelines — or maybe you're doing some administrative task. They're going to be talking to your Nomad daemon that's running in your cluster somewhere. Then, when you do a job spec, or you do a change or something like that — or a pipeline comes through — it will talk to your Docker daemon. The Docker Daemon is then going to talk to your load balancer and set up your new web app.

We're using the Fabio load balancer in our case, so I labeled it that way — in the Nomad way of thinking and terming, we call that a task group. It'll have your database and your daemon as a job spec. 

The whole thing will be talking with Consul on the side. So Consul is going to be interacting with your web apps. It will be giving them health checks and making sure they're healthy. If they're not healthy, it will let the load balancer know they're not healthy.

Let's say you have five deployments —and one of them is not answering very well — all your traffic's going into the forest. That keeps everything nice and clean. This is a perfect little environment, and it's so perfect. It makes you happy, and it makes your customers happy. So simple — so straightforward. I love this stack. It's my favorite stack.

If you want to see that whole demo that we did, you can go to this URL. If you want to see the full project.nomad — you want to see the little goodies and things you can include and customize, or maybe you want to take it over yourself —Or maybe you want to copy it — that's totally fine. It's right here on the link.

»Customization

If you want to customize things — let's say you want to have a custom hostname or change the count or number of deployed things —‚ it deploys one by default. Let's say you wanted to deploy ten — so ten things that are running all get automatically load balanced. You can use nomad_var_count, nomad_var_hostnames, and things like that. You can add any of these to your gitlab-ci.yml, as little variables up at the top, and those will get included and customize your setup.

»Issues With Docker Login?

We run everything on-prem at the archive, by the way. We have our own GitLab, we have our own Nomad clusters and Kubernetes clusters. We're mostly moving to Nomad at this point. If you have issues with Docker login — we did — you might want to consider creating a read_registry deploy token. You can set those as CI/CD variables that will then pass right through into the job spec. We're using CI R2 user and CI R2 paths — which sounds like a droid — hopefully, we don't get a lawsuit — but that will help you out there. 

»Monitoring and Usage

All of our devs are using these nice little aliases to make it easy. So whether you're on the command line or whether you're in vs code, they will look at your context or your current working directory and do some Git information. You can type nom-ssh, and it will SSH you right into the container related to the code base you're looking at — or the branch you're looking at. nom_cp will hot copy a file into the container, so you don't have to do a full pipeline if you don't like — that's super-nice. I don't know why everything doesn't have that. 

Then if you want to do something like node status or detailed information — or some of these other aliases — they're all there in that link above. If you do Docker builds, these two things took me two years to figure out. I view it as my responsibility to let you know — so you will know as well — if you don't know. 

If you’re using GitLab, I recommend a single GitLab CI/CD server — and not using docker-in-docker. Use docker.sock. That way, you can build up your cache over time and through multiple deployments. Otherwise, docker-in-docker starts with blank, blank, blank, and you lose all the caching.

That second link below — GitLab runners and monorepos — it's super-good. If you do builds, look at that link. I wish I had found that earlier because if you have big repos that are running through, it will be very hard to get good CI/CD times without something like that.

»Additional help

If you want to see how we set up our clusters with this relatively simple shell script with no dependencies, we can set them up on Linux on DigitalOcean VMs. You can do them as little as $5 a month or $20 if you want the whole shebang with GitLab and GitLab runners. We can also run it on your Mac or laptop. We've got a script as well if you need HTTPS certs with let's encrypt for domain level. If you want to see some pipeline examples, there's a link here.

»Results 

We moved to Nomad about a year ago — after a stormy month with Kubernetes especially and talking with our ops people who already knew and loved Consul. We’ve moved over 70 repos — or Recode repositories or web apps. Over 400 review apps have been deployed. Our deployments are now twice as fast, and all of our main development is using GitLab and Nomad review apps, which is awesome.

»Takeaways

Deploy with GitLab and Nomad. Two-line GitLab-ci.yml. Two CI/CD variables. That's it — you'll be off to the rocket races. Go, go, go!

This is the end. That's not my cat. Hopefully, this cat was not hurt in the making of this video. That's very cute. I want to thank GitLab and HashiCorp for this opportunity and for working with us in the industry and vice versa. It's a great relationship. Thank you for attending and have a great HashiConf Europe.