Waypoint Updates & Roadmap from HashiConf Europe 2021

This talk shares Waypoint’s vision to add ease back into the development flow. We will cover exciting new features and share current best practices.

Speaker: Rae Krantz

»Transcript

Hello. It's great to be hanging out with you all today at HashiConf Europe. As you know — through the magic of time travel — I am in the chat with you while I'm giving this presentation, so we're going to have some fun throughout this talk. 

My name is Rae. I've been on the Waypoint team since the launch in October. And I’ve been at HashiCorp for a little over two years. I'm very excited to work on a product that is solving some of the pain points that I helped development teams with over the years. 

How to find me: I use different handles on Twitter, GitHub, and Slack, and in every mobile game I've ever downloaded because I'm not cool enough to have one consistent gamer tag everywhere. But on Twitter, I am RustbeltRae. That name is from the Rust Belt in the US — where I lived for about a decade — and it will always feel like home. I will happily answer questions about the Rust Belt for fellow geography and etymology nerds. But for now, let's move on to the hero of our story, Waypoint. 

»Introducing Waypoint

Waypoint was built because dev teams want to deploy their applications, and ops teams want to enable that smoothly. Waypoint gives ops teams a place to define the path that an application should follow from build through to production— and then provide that map to dev teams to use in their everyday flow. 

This sounds simple enough. But we all know that reality takes a lot of string and tape to move the code in your repo through the build deploy release process: A lot of custom-built in-house deployment systems covering a lot of platforms and passing around data and artifacts — before eventually pushing that application that we built onto our release platform. It sometimes even involves different custom-built tools across teams in those very same organizations.

Waypoint exists so your ops and dev teams have the same tool they can use to deploy across platforms, across teams, and across products — making deploying easier and onboarding quicker. We do this through the waypoint.hcl. This gives your development teams access to one central command, waypoint up, — and your build deploy release can run backed by whatever platforms you're already using and whatever tools you already know and love. 

»Installing Your Waypoint Server

Let's install our server. This is the first thing that you would do if you were going to get Waypoint up and running. And right now we're going to install it on Nomad. Installing the server is a one-time event. Then you can share access to that server with your dev teams via Waypoint's token system. 

Once we have our server running, we can initialize our project — which is registering our project with the server — and run waypoint up. A project maps to a VCS repo and contains one or more applications. Running up — as you see here — is a combination command. It will run the build, deploy, and release commands within Waypoint in sequence. We have plugins for each stage as the Waypoint ecosystem is fully extensible. So you can make your own plugins that you may need for your system and your platforms. 

»The Pack Plugin

One such plugin that you see here is the pack plugin, which we are using to rely on the build pack system to build our artifact that we are going to deploy. This is fun because if you don't already have a Dockerfile — or something like that — you don't need one. 

Once we have our artifact built — that will happen in the next few seconds — you will see it get passed to our deploy stage and then the release stage. Waypoint, right now, we are running a URL service to provide you with deployment URLs that you can use to test your application once it's deployed. 

The release URLs are platform-specific. Mine, you'll see that the release URL and deployment URL do have the same host name of Arguably Credible Kite. That's because I'm running my Nomad node locally. But if you were to deploy or install your server on — say a cluster in AWS for example — then that release URL would be supplied by that particular platform. 

A lot of people ask, “Hold on, how does Waypoint fit into CI/CD? Does it replace the whole thing?” The answer is no. But Waypoint certainly does fit into the setup that you currently have going for you — and possibly replaces a lot of lines of Bash and other scripts that you might have calling one tool from another. Here are a few examples that we have seen already, both in the community and from our own team. Check them out and let us know what other use cases you come up with. 

Alright, let's go over briefly what Waypoint does now — or did — when we launched in October as the MVP. Then we'll move into talking about some of the newer features we've been working on.

»What Waypoint Brings

As a base use case, we provide the waypoint exec command for exec-ing into deployments and running things like database migrations. logs, which allow you to tail active deployments — those preview URLs that we talked about. We do use TLS by default for this. And, of course, everyone's favorite random pet generator. 

A bonus of this is that you learn things. This particular URL led us on a journey of discovery, learning that Dassies are also called Rock Hyraxes. Their closest living relative is the elephant, and they live in packs of 10 to 80. Speaking of working in teams, Waypoint makes this easier too. Application lifecycles are managed on a central server which you can provide your dev teams access to through our token authentication system. You can roll back to previous deployments, view your application history in the UI, and set application configuration within the waypoint.hcl.

»Example Waypoint UI

Here's an example of our UI, showing how Waypoint offers a link between dev and ops. You install the Waypoint server on central infrastructure, and each developer can build and deploy applications. The server centralizes development on the team across projects. Anyone on the team can see the status of a given project and application, deploy new versions, exercise rollbacks, and monitor via logs and other features we've added recently and plan to add in the future. 

Ops teams now have a way to provide a smooth workflow to dev teams from a unified platform. Waypoint is fully extensible.o You can fit together whatever pieces are a part of your team's puzzle then give them an interface to take your apps from development to production.

»What’s New? 

Like I mentioned, the October launch of Waypoint had all of the above as part of that MVP, but we have been doing a lot of work since then. Let me tell you what that has included. 

»Dynamic, Synced Configuration

Building on top of application configuration variables, we've added dynamic config values that sync automatically with your applications. The example here shows how this might work with a Kubernetes ConfigMap. The sync actively watches the defined source, updates the environment variable whenever the value changes and restarts your application. 

To give you a bigger picture of the impacts of this functionality, here's an example of a config stanza in the waypoint.hcl. As you can see, dynamic values make it easy to use external configuration stores that may have sensitive or regularly changing values, such as Vault or Amazon Parameter Store. 

There are a lot of variables your application needs to run, and you don't want to have to copy that from Vault to the server — for example — or write some weird bash script to take your TFC outputs and give them to your application. Now you can give your application the values it requires to run without storing them on a developer's machine or the server. We currently have several plugins that utilize dynamic configuration, including the Terraform Cloud plugin that you see mentioned here. I encourage you to check those out. 

»Waypoint Health Checks

Up until now, the waypoint.hcl verified that the artifact was successfully deployed. Some platforms may require additional time before something that is running is ready to receive traffic. Waypoint's health checks use health report forwarding, giving you quick visibility into the liveness of your applications from the waypoint.hcl and UI. 

This reporting uses the existing health checks that you have already configured on the platform and surfaces the reports, so it's easier to view them across your deployments and releases. Previously, Waypoint would let you know if a deployment succeeded or failed at the time of deployment. Waypoint up, here you go. You got a deployment URL, it worked. But now we can also pull the health checks of the platform, providing you up-to-date information and giving you confidence in your pipeline's health. 

We clearly show you if something is running that's still starting up, up and ready for traffic or down per that specific platform’s own health check interface. Now dev teams don't have to know commands for health checks for each of the myriad platforms you may use in your deployment process. Your ops teams can configure that and your dev teams can access everything through one central interface —Waypoint. This feature also enables some exciting future development that I'll talk about later.

»GitOps

GitOps enables you to create a project, configure, Git repo settings, and tell Waypoint to pull that repo for changes entirely from the web UI. You may be very familiar with functionality like this in other tools that you're already using. This is a powerful feature that can be combined with some of our existing plugins to power up your workflows.

So here in this demo, you'll see us authenticate to the UI with a server token and then create a new project using some basic Git settings. Our UI interface allows you to set your Git credentials if you need them. We won't be using those here because it is an open source repo. And, of course, you also will be able to set up GitOps without needing any authentication, if you wish. 

When we first set this up, it triggers an immediate poll to initialize the project. That's what you see running here. Subsequent builds would run off of updates to the Git repo. The application that is being spun up is a small Ruby app running on Lambda that queries the URL parameters to update the app.

You'll see a quick example of that here as we visit the deployment URL. And then, because this is Waypoint, we have access to logs for our application. Those are now showing up and we are able to look at how our application is running. 

»Mutable Deployments 

Next, let's talk about mutable deployments. Waypoint's base functionality creates resources for each deployment. Every deployment is a whole new thing. However, not all platforms that you may use mesh well with this model, so we have added the concept of mutable deployments. 

This functionality is fantastic. Not only because it allows you to update your deployments in place when you want to deploy a new build. But because, now if you want to try out Waypoint with your existing flow, you can give us your Nomad job file — or Kubernetes YAML — and Waypoint will know how to create deployments from them using those plugins.

As you can see, in this example we are using the Nomad-jobspec plugin to pull in a templated Nomad job file, which can itself reference point output variables, such as the artifact from the build stage. Running waypoint up here will publish the deployment to the running Nomad job, rather than removing the current job and creating an entirely new one. 

We can see another example here with the Kubernetes apply plugin. This works similarly to the example we saw in that you can use a templated version of your Kubernetes resource file. When a new waypoint up is run, we build the new image and then deploy it in place using the existing config. You can see in the output for the release stage, it reads that this deploy was done in place. That's what you'll see if you do in-place deployments.

You'll also notice that the deployment URL still says V1. That's because rather than creating a V2, V3, etc., deployment — as you may typically see in a base Waypoint use case — we are updating in place. So that deployment reference remains the same within Waypoint as well. 

»Server Snapshots

Finally, this is not the flashiest feature. But it is definitely a foundational one. And we are excited that it is a core part of Waypoint. Server snapshots and restores. We've built server snapshots and restores into commands, such as the server uninstall and server upgrade commands, so that if anything goes wrong with running those, you have a snapshot that you can revert to. And it is there for you to use through the CLI as part of your current backup process.

»What’s Next?

This was some of what we have done since October. It's a lot, and it's all exciting and there's more that I didn't quite have time to go into. But if you like, there are lots of Learn guides and documentation on our website. And the team is hanging out in our Discuss board and GitHub issues page. We're eager to hear what you think of what we've built so far and what's coming up. Speaking of what's coming up, we are actively planning our work for the coming year and want to share a little bit about what that might include. 

»Resource Awareness

As I mentioned earlier, the work we've done to support health checks will allow us to add even more observability into Waypoint deployments. We want to give you more insight into the status of your applications resources. 

»ODIC Authentication 

To expand on our current token system, we want to implement patterns we've used in our other products to give you more governance of users who are authenticated with your Waypoint server. 

»UI Enhancements

We've tripled the size of our UI team from one to three and are excited for the ideas that they've slated to work on. We will bring more parity between the commands you can run in the CLI in the UI and strengthen the use cases for workspaces in Waypoint. 

»Workspaces

Speaking of, Waypoint has workspaces. But there is a lot more they could be doing. We'll be expanding our story around workspaces and making them much stronger parts of the Waypoint story. 

»Learn Guides

Finally, those Learn guides I mentioned? We're going to continue to build those out. We know that one of the most important things about a new tool is that you know how to use it and you know all of the cool and fun things that it can do so that you can use it to its fullest extent — so keep an eye out for those. 

There's a lot more, like I said, that I didn't quite have time to show you. But I encourage you to keep an eye out on our blog and emails for announcements. Specifically, I encourage you to check out the videos from HashiTalks 2021. Nomad's Michael Lange shows how you can release the same application using Kubernetes and Docker, Nomad and Docker, and Nomad and raw binaries all with Waypoint (The talk is: Kubernetes to Nomad & Containers to Binaries with Waypoint). And there are some great talks from other folks in our community too. 

It's been fun talking with you, and I hope to see you around our Discuss board and GitHub repo. What can you deploy with Waypoint?