HashiCorp Waypoint: A Whiteboard Video with Armon Dadgar

For some time, we've thought about whether we could bring a simplicity to the build, deploy, and release steps for developers, similar to the simplicity of virtual environment setup that we brought to developers with HashiCorp Vagrant. Now, we think we can.

HashiCorp Waypoint is aimed at giving developers a consistent workflow to build, deploy, and release applications across any platform. Using a single command: waypoint up, developers can get their applications from development with only the source code and a manifest file for Waypoint.

Developers will also be able to have confidence that the deployment is working with logging and debugging tools, and the integration process will be infinite given Waypoint's extensible plugin architecture.

Transcript

I want to spend some time talking today about one of our newest projects, HashiCorp Waypoint. But before we jump right into what the tool is and how it works, I want to set a little bit of context for why we created the tool to begin with.

As you know, we're a maker of a whole bunch of tools. Everything from Vagrant, Packer, Terraform, Consul, Vault, Nomad, and now two more tools, Boundary and Waypoint. So it's a natural question that we often get asked, which is, "There's so many tools out there, there's so much complexity in the modern ecosystem, what's the right interface for developers? What do developers use to deploy and manage the lifecycle of their applications?"

Breaking Down the Application Lifecycle Steps for a Developer

This ecosystem has only gotten more complicated over the last few years with containers and Kubernetes and serverless, etc, the list goes on. When we step back and say, "Well what are the core lifecycle steps that a developer really cares about?"

Write

It starts with, they want to write their code. If they're writing their application, they're probably doing this in an IDE, their editor of choice, VSCode, Vim, Emacs, etc, the list goes on.

Test

Then the next step is, how do I test that application? Probably one of the most common patterns would be integration with the CI/CD system. So you commit to GitHub, as an example, and then you run your tests in CircleCI (one option for continuous integration), and you have an automated set of tests that run every time you commit.

Pushing to Production

Now I think where things start to get challenging is the steps beyond that. Which is: I've written my application and I tested it, but now I want to push that out into production. I want to deploy this application. When we talk about deploy in a broad sense, there's usually a few sub-steps in there.

Build

One is how do I build my application? If it's a containerized app, how do I build my Docker container and push it to a registry? If it's a serverless app, how do I compile it to the appropriate format that my runtime understands? Maybe it's a VM-based application and I need to bake a machine image or an AMI (Amazon Machine Image), etc, but there's a build step where I need to transform from source code into something that's runnable, and I need to put that runnable artifact into a registry and keep track of it.

Deploy

Then the next step is after I've done that: I need to deploy. And here, what I really mean is mapping of that artifact that we've created onto the runtime. This might be uploading that ZIP file to our serverless environment. It might be pushing our new Docker container out to my Kubernetes cluster. It might be spinning up a new VM based on the image we just created, etc.

Release

After we've done that, how do we manage a release? The difference between a deploy and a release being; maybe I have version 2 of my application running, I deploy a new VM running version 3, how I do the cutting over so the traffic starts to go from version 2 to version 3? You can think about release as being focused on that traffic management, the traffic shifting, and here's where we might apply patterns like a blue-green or canary deployment.

Operate

Underneath all of this, there's a set of tooling that we use to operate. Here's where our operational infrastructure sits. This is where I'd put HashiCorp Terraform, Vault, Nomad, and Consul. And Kubernetes and OpenShift, etc. These are the platforms we use to operate our infrastructure, and typically they're run and provided for developers by the ops teams internally.

Measure

On top of all of this, how do I measure or observe what's happening? I might use Splunk for my logging. I might use Datadog for my telemetry or my APM, and I have tracing solutions, etc. There's a whole set of systems I'm going to run that'll provide me that visibility, that observability into how the system's operating.

One Workflow for Build, Deploy, Release

The developer concerns are greatest on the left and diminish as we get to the right. And certainly the responsibilities diminish as we get further to the right, so our view is there's a very well-established pattern around how we write and develop locally. Developers know and love their IDEs, their dev tools of choice. There's great patterns around continuous integration and having our test servers. It's this middle area (build, deploy, release) that we felt like there was the greatest problem.

How do we provide a consistent and simple experience to developers in this middle, regardless of what their technology looks like? What I mean by that is: yes, we can train everybody on how you do it specifically—to deploying a VM image onto EC2, and here's a different way of doing a Kubernetes-based deploy with Docker onto GKE. And here's a third way of doing your JavaScript function as a Lambda onto Amazon, but there's an enormous amount of complexity there. So our goal, our view is: how do we simplify that? So that as a developer, what we focus on is two things.

1. Our source code of the application.

2. A lightweight manifest. This includes some metadata that our application needs. It's a bit of a declarative description above and beyond what would live inside of the source code. Maybe it's what region we need to deploy in, or how many instances we need, etc, things like that.

And then these two should functionally be the input to the platform layer. Everything below this platform is plumbing. We don't care if this is deploying onto my Nomad cluster on-premises. Is this deploying onto my Kubernetes cluster in Google? Is this deploying onto my Lambda environment in AWS? In some sense, it doesn't matter to the developer. What we want is a consistent workflow on the northbound that abstracts these details from us. I want a consistent way to build, deploy, and release my app regardless of what the technology is that's being used underneath.

Waypoint

This problem is exactly what Waypoint was designed to solve. The number one goal with Waypoint is to act as this layer. This is where Waypoint sits. The goal is: alongside the source code, you'd have your Waypoint configuration file. That thing declaratively tells the system, how should I handle this application? What are its additional requirements? But shielding the developer from these specifics. The goal is, once you've gotten past your writing, you've tested your app, you're ready for this phase of build, deploy, and release and you want to get out to production. It's a single command line.

The only thing I need to invoke is a waypoint up.

What waypoint up will do is basically consult the manifest—the declaration of how the platform is configured—and then it will go and execute the build, the deploy, and the release sequentially based on that configuration in a way that might be unique to the different platforms.

Build

The way we have to do build for a machine image might be: I have to go bake a VM image versus I might have to do a Docker build if it's a container.

Deploy

The way I might do a deploy might be based on talking to Kubernetes if that's the environment I'm deploying onto. It might be using Terraform to launch a set of VMs.

Release

When we talk about release, that might be talking to an Amazon load balancer, it might be fiddling with Consul to change the way traffic is managed, but it's to allow that release management to happen in a way that's transparent to how they use our things.

The goal is around this consistency of experience. I want to shield my developer and team, to the degree possible, from what the details of the plumbing are and give them a single simple experience. Of course, if you wanted more power than just a single command line that did all of this, you could explicitly run a waypoint build, a waypoint deploy, a waypoint release etc, and get more involved. But at the highest level, it's the single command, this one north star for getting any of this deployment done.

Waypoint Plugins

That solves one piece of this problem, which is how do we abstract away some of this plumbing, but the reality is this stuff is complicated. I have all sorts of different choices, different platforms, different things I need to integrate with. So a core tenant of Waypoint is making it very, very integratable. Much like Terraform has providers that can be plugged in, Waypoint has plugins for all of these different pieces.

So you can have plugins that define how a build should take place. Plugins that define how an artifact should be stored, whether it's in Artifactory or a different system. You can have different deployment methods. So this would allow it to target everything from Kubernetes to VMs, to serverless environments, etc in different systems you might be using for release management or traffic management. So all of these are different plugins so Waypoint is designed to be hyper-extensible.

A Waypoint with GitOps Example

The other side of this is really acknowledging that there's multiple ways developers want to work. One way people want to work is through a GitOps approach: where I'm going to take my source code and I'm always going to commit it to version control system. Might be GitHub, might be GitLab, etc. And I'm going to use a convention where I say, for example, only when I push and change to master branch—or maybe I have a production release branch or a staging release branch—that will get picked up and pushed out to the production environment or staging environment, etc.

So I'm going to tie a strong notion of the Git semantics into how I want my deployment process to work. And that's great. You can integrate Waypoint with this GitOps model. In fact, we have GitHub actions available day one that allow you to do exactly this. So you can tie into that and say, great I'm going to invoke Waypoint automatically to now go do that release based on this model. Here we might be invoking Waypoint based off of that.

Waypoint in a ChatOps Model

Another common model we see is a ChatOps model where I'm going to have my developers—they're actively committing, maybe we don't have a strong convention around how we're doing our version control, we're just committing and using a main line to keep track of the latest version and keep everyone consistent. But at periodic points we decide we want to do a release based on a particular Git SHA or tag etc. Here Waypoint can be integrated with systems like Slack, so you can interact with it like a chatbot. So as a user, my primary interface might be a system like Slack or HipChat, where I'm using a textual interface to trigger doing a deploy, which is ultimately interacting with Waypoint as well.

Waypoint CLI and UI Workflows

The third methodology is: what if I don't want to set up all this complexity? I'm just getting started, it's a small team, I just want to focus on my application and be able to iterate rapidly on it. I don't need to GitOps system, I don't need a ChatOps system, I just want to be able to quickly deploy from the command line.

That's great. You can directly use the CLI and run Waypoint locally. Just run a waypoint up. You don't need to integrate it with GitHub, you don't need to integrate it with Slack, you can just execute it locally. In addition to that, Waypoint has a full fledged UI. So you could run the Waypoint UI locally, use that to inspect your builds, your deploys, your releases, see how things are going.

The goal is for Waypoint to meet you wherever you're at. Whether you're two people just getting started and you want a simple experience that hides a bunch of the platform complexity, or you start to operate in a larger setting where you have conventions like ChatOps or GitOps. Those can be integrated as well.

How Waypoint Gives You Confidence that Things are Working

The other piece that we haven't talked about here is a classic a problem. I committed to my version control system that triggered and did a deploy, that's asynchronous. I did my git commit and the deploy took place sometime after that. How do I know, as a developer, that is this thing working?

Did I deploy a build that was broken and I'm getting a bunch of 500 errors in production now? How do I interact with it? And if I need to debug something, how do I quickly get an environment that lets me do so?

Waypoint Logs

The first approach here, and this is all part of a broader goal of: how do we bring confidence to the end user? Much like Terraform has the plan command that gives you confidence that you know what the tool is doing, with Waypoint the goal is: how do we give you confidence that you understand what's happening in production?

Right after you finished doing your deploy, you can run the waypoint logs command. And you'll quickly start to get a streaming set of log output from the running instances of your application. This works regardless of what platform you're operating on. Whether you're deploying to Kubernetes, whether you're deploying a Lambda function, whether you're deploying a VM, you can run Waypoint logs and instantly see the most recent log output and a streaming output coming from the instances that were deployed.

This is not meant to be a replacement for your central logging cluster. It won't let you go and search back years of history, but it's meant to be able to answer that question: I just did a deploy, show me the last 30 seconds and show me what's happening right now. Help me have confidence that I understand that the system is working.

Waypoint Exec

The second thing we have available is waypoint exec. What waypoint exec will do is drop you into a production session where you basically have a shell available. So you can run any command you need to here. What this might be useful for is everything from:

• Great, I just did a deploy, I need to do a schema migration, so I need to connect to my database and do a schema migration.

• Maybe my application is stuck and I want to debug it, and I want to get an environment where I can interact, and it looks like production.

There's a ton of different uses. In general, I think there's a broader movement that says we shouldn't have SSH access directly to production. We shouldn't be executing things directly in prod. And I think that's generally true. To the degree possible, we want to use automation. We want to push this stuff so that we're not depending on manual human interaction for this stuff. But we all acknowledge there's pragmatic limits to that.

If I'm just getting started with a project, I don't want to set up a whole bunch of complex automation just to do my schema migration. If something's truly broken and I need a break-glass way of debugging in that environment, this is a valuable tool to have in the tool chest. So like all things, they shouldn't be abused, but it's a valuable thing to have and it's part of trying to build that operator-developer confidence that we understand and can debug these systems.

Waypoint Goals

At a very high level, the goals of Waypoint are threefold:

Reducing Complexity

Looking at this a challenge—there's an explosion of complexity in the the southbound, the platforms that we're operating on—how do we give simplicity back to the end user? How do we bring it back to something like waypoint up where we don't have to know the nuances of how a build, how a release, how a deploy takes place across different platforms, across different runtime environments, and bring much more sanity to the developers who are trying to interact with this system and focus on the application value.

Unopinionated Extensibility

Acknowledging that there is so much different technology that you might want to integrate with that you already have in your ecosystem. Waypoint isn't a platform. It's not hyper-opinionated and says you must use this runtime, you must use this packaging format. It tries to be prescriptive about a simple workflow for developers, but it's designed to be highly pluggable and highly extensible so that you can bring whatever infrastructure you already have, whatever platforms you already have.

In the same vein, it's not hyper prescriptive about the methodology. If you want to use a GitOps type approach and tie it into your version control system, that's great and there are integrations. If you want a ChatOps type integration, where you interact with it in a more interactive way, that's great. If you want more of a Click-Ops approach and you want to interact with the CLI and the UI, that works as well.

These aren't exclusive. You might say 95% of my time, I drive it on GitOps, but when something goes wrong, I have that portability that I can execute Waypoint outside of that environment. Potentially because I need to do a rollback or I want to debug a deploy that didn't happen to work correctly. Can I have a workflow that's portable outside of my CI environment? Or outside of my ChatOps environment when something goes wrong? These aren't exclusive, you can mix and match and use whichever makes most sense for your team.

Confidence in Production

The last one is this notion of confidence. I do a deploy. Do I understand what's happening? Can I quickly debug it and have confidence that my deployment worked?

Waypoint in a Nutshell

At a very high level, this is Waypoint in a nutshell, focusing on that build, deploy, release lifecycle and how we can make that super simple. If you're interested in learning more, please check out the website. There are a lot more resources available there. Thanks.