Waypoint 0.10 Brings Custom Pipelines and Nomad Plugin Updates
HashiCorp Waypoint 0.10 — now generally available — introduces the tech preview of custom pipelines and improvements to the Nomad integration.
We are excited to announce the general availability of HashiCorp Waypoint 0.10. Waypoint is an application deployment tool that aims to deliver a Platform-as-a-Service (PaaS) experience for Kubernetes, Amazon ECS, HashiCorp Nomad, and other platforms. Waypoint 0.10 adds several significant new features, including the tech preview of custom pipelines, a new CLI command to tear down existing projects and their created resources, as well as many enhancements to the Nomad/Waypoint server and deployment plugin integrations.
»Custom Pipelines (Tech Preview)
Up until this release, Waypoint had a relatively fixed built-in pipeline with the waypoint up CLI, which ran a build, deploy, and release for an application. An up was a special operation that encapsulated the build, deploy, and release process.
For most users getting started with Waypoint, this simple pipeline is powerful enough to get off the ground continuously building and deploying an application to production. However, as projects and teams become more complicated to address real world concerns, users need more control and customization over the order in which Waypoint deploys applications — running tests after a build, for example, or security scanning artifacts prior to a deployment. This level of control is not easily achieved with the current Waypoint model.
Custom pipelines are an answer to this problem. This tech preview lets users define pipelines that can run in whichever order is required to deliver an application. Additionally, custom pipelines allow for user-defined execution steps through the exec plugin that will run any kind of bash script as part of the pipeline execution for such operations as running tests or security scanning.
Custom pipelines let users tell Waypoint what order these operations should be run in, as well as indicate any operation options that might need to be configured, like disabling an artifact push for the build step or pruning old deployments on a release in the release step. Waypoint projects still expect the application to be defined outside of a pipeline in its own app stanza.
Here is an example that redefines the up operation to be a pipeline with three steps for a build, deploy, and release. This example pipeline could be used as a starting point for more complicated pipelines.
project = "my-project"
pipeline "full-up" {
step "my-build" {
use "build" {
}
}
step "my-deploy" {
use "deploy" {
}
}
step "my-release" {
use "release" {
}
}
}
app "my-app" {
build { ... }
deploy { ... }
release { ... }
}
For the exec steps, it accepts a command and arg parameter for running commands.
pipeline "build-and-test" {
step "my-build" {
use "build" {
}
}
step "test" {
image_url = "https://localhost:5000/waypoint-odr:dev"
use "exec" {
# executes a binary test with some arguments
command = "test"
args = ["--full", "test-all"]
}
}
step "my-deploy" {
use "deploy" {
}
}
}
To run the full-up custom pipeline, it's as simple as waypoint pipeline run full-up:
brian@ubuntu:waypoint-tetris λ waypoint pipeline run example
» Running pipeline "example" for application "tetris"
✓ Pipeline "example" has started running. Attempting to read job stream sequentially in order
✓ 3 steps detected, run sequence 2
Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"
» Cloning data from Git
URL: https://github.com/briancain/waypoint-tetris.git
» Downloading from Git
Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
Timestamp: 2022-08-29 17:04:04 +0000 UTC
Message: Adding example pipeline
» Executing Step "build"
Reading job stream (jobId: 01GBNAQRW2M6TTZ4C0JKT0AXRD)...
Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"
» Cloning data from Git
URL: https://github.com/briancain/waypoint-tetris.git
» Downloading from Git
Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
Timestamp: 2022-08-29 17:04:04 +0000 UTC
Message: Adding example pipeline
✓ Running build v1
✓ Building Docker image with kaniko...
✓ Testing registry and uploading entrypoint layer
✓ Executing kaniko...
│ INFO[0001] Retrieving image manifest nginx:stable
│ INFO[0001] Returning cached image manifest
│ INFO[0001] Executing 0 build triggers
│ INFO[0001] Unpacking rootfs as cmd COPY ./public/ /var/www requires it.
│ INFO[0004] COPY ./public/ /var/www
│ INFO[0004] Taking snapshot of files...
│ INFO[0004] COPY ./nginx.conf /etc/nginx/conf.d/default.conf
│ INFO[0004] Taking snapshot of files...
│ INFO[0004] Pushing image to localhost:32953/tetris:latest
│ INFO[0005] Pushed image to 1 destinations
✓ Image pushed to 'my-container-registry/tetris:latest'
✓ Running push build v1
» Executing Step "deploy"
Reading job stream (jobId: 01GBNAQRW29Q1M7QNQ0F1JX2CS)...
Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"
» Cloning data from Git
URL: https://github.com/briancain/waypoint-tetris.git
» Downloading from Git
Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
Timestamp: 2022-08-29 17:04:04 +0000 UTC
Message: Adding example pipeline
✓ Running deploy v1
✓ Kubernetes client connected to https://10.96.0.1:443 with namespace default
✓ Creating deployment...
✓ Expected "http" port "3000" for app "tetris-v1"
✓ Deployment successfully rolled out!
✓ Horizontal Pod Autoscaler has been created
» Executing Step "release"
Reading job stream (jobId: 01GBNAQRW22N7PQXSF2KGXT1MS)...
Performing operation on "kubernetes" with runner profile "kubernetes-bootstrap-profile"
» Cloning data from Git
URL: https://github.com/briancain/waypoint-tetris.git
» Downloading from Git
Git Commit: 2133658e209f623c019ddb3c9e48c2c7ee9f4d1f
Timestamp: 2022-08-29 17:04:04 +0000 UTC
Message: Adding example pipeline
✓ Running release v1
✓ Kubernetes client connected to https://my-kubernetes.example.com:443 with namespace default
✓ Creating service...
✓ Service is ready!
✔ Pipeline "example" (k8s-tetris) finished! 3/3 steps successfully completed.
Each time a pipeline is run, Waypoint keeps track of the steps and run results. As you can see from the example waypoint pipeline run console output, Waypoint identifies how many steps are in the pipeline and how many are successfully completed. Users can then deep dive into specific pipeline runs to view more information.
waypoint pipeline list shows the current steps, last run start and complete times, last run state, and total runs for every pipeline in the project:
❯ waypoint pipeline list
» Waypoint Pipelines for my-project
ID | NAME | OWNER | CURRENT STEPS | LAST RUN STARTED | LAST RUN COMPLETED | STATE | TOTAL RUNS
-----------------------------+------+--------------+---------------+------------------+--------------------+---------+-------------
01GBBBC63TR5DYQXQVGJPS8JGZ | test | example-java | 2 | 5 days ago | 5 days ago | success | 5
waypoint pipeline inspect can look up a specific pipeline by ID (default) or name, as well as an additional option to view a specific run of that pipeline. Pipeline run information will include run state, steps, timestamps, and associated job IDs.
❯ waypoint pipeline inspect 01GBBBC63TR5DYQXQVGJPS8JGZ
OR
❯ waypoint pipeline inspect -name=test
» Pipeline Configuration
ID: 01GBBBC63TR5DYQXQVGJPS8JGZ
Name: test
Owner: example-java
Root Step Name: build
Total Steps: 2
Total Runs: 9
Last Run Started: 1 minute ago
Last Run Completed: a long while ago
Last Run Status: RUNNING
```
```
❯ waypoint pipeline inspect -name=test -run=3
» Pipeline Configuration
ID: 01GBBBC63TR5DYQXQVGJPS8JGZ
Name: test
Owner: example-java
Root Step Name: build
Total Steps: 2
Run Sequence: 3
Jobs Queued: [id:"01GBBBQJZHZVMKRAQSXGYXKEZY" id:"01GBBBQJZH05YD6MN91X30G7HF"]
Run Started: 5 days ago
Run Completed: 5 days ago
State: ERROR
Additionally, pipeline steps can be scoped to a workspace for executing tasks in various environments. By default, the entire execution of a pipeline is performed in the same workspace used when performing the run, or default if not specified. By setting the workspace attribute for a step, users can execute a specific step in a specific workspace.
This example pipeline could be used to build and deploy both a test and staging version of an application and deploy them to their respective workspaces:
project = "my-project"
pipeline "test-and-staging-up" {
step "test-build" {
use "build" {
}
}
step "test-deploy" {
use "deploy" {
}
}
step "staging-build" {
workspace = "staging"
use "build" {
}
}
step "staging-deploy" {
workspace = "staging"
use "deploy" {
}
}
step "default-release" {
use "release" {
}
}
}
app "my-app" {
build { ... }
deploy { ... }
release { ... }
}
We plan on continuing to make improvements to custom pipelines in the coming months to bring the capability out of tech preview. This includes more support for running batches of operations for a pipeline by improving Waypoint’s internal job system, sharing VCS project source code between steps, giving custom pipeline steps the ability to share input and output variables, approval steps to "manually" sign off on releases, and more!
Given that custom pipelines are a tech preview in this release, we invite all Waypoint users to give the feature a try. Let us know about your experience with it, share your feature requests and enhancements, and note any bugs you might find via our Waypoint Discuss forums or on GitHub.
»Project Destroy
Waypoint has always allowed users to create and update projects, but never remove them. The new waypoint project destroy command will delete your project from Waypoint.
$ waypoint project destroy
Do you really want to destroy project "kubernetes-nodejs" and its resources? Only 'yes' will be accepted to approve: yes
» Performing operation locally
» Destroying releases for application 'kubernetes-nodejs-web'...
✓ Running release destroy v1
» Destroying deployments for application 'kubernetes-nodejs-web'...
✓ Running deployment destroy v1
✓ Kubernetes client connected to https://kubernetes.docker.internal:6443 with namespace default
✓ Deployment deleted
Project "kubernetes-nodejs" destroyed!
By default, the waypoint project destroy command will destroy any deployments that have been deployed for apps in your project. This ensures that the resources deployed to your platform of choice aren’t left behind after Waypoint is no longer managing your app. You can opt-out of the destruction of these resources using the -skip-destroy-resources flag. This results in Waypoint simply deleting your project, its apps and their builds, artifacts, deployments, releases, and status reports from the Waypoint database. Pipelines, triggers, config variables, triggers, and workspaces for your project will also be deleted.
»Nomad/Waypoint Integration Improvements
With the recent release of Nomad 1.3 came Nomad service discovery, a service discovery solution built into Nomad without the advanced features of Consul. Waypoint 0.10 takes advantage of this new Nomad feature by allowing the service discovery provider to be specified for the Waypoint server. Consul may still be used, of course, but if you’re learning how to use Nomad and Waypoint together, and haven’t yet moved on to designing a service mesh, using Nomad service discovery makes installing the Waypoint easier. This builds off of changes made in Waypoint 0.9, which enabled the use of Nomad service discovery for deployments with the Nomad plugin.
$ waypoint install -platform=nomad -nomad-host-volume=waypoint -nomad-runner-host-volume=waypoint-runner -nomad-service-provider=nomad -accept-tos
✓ Waypoint server ready
The CLI has been configured to automatically install a Nomad service for
the Waypoint service backend and ui service in Nomad.
✓ Configured server connection
✓ Successfully connected to Waypoint server in Nomad!
✓ Server installed and configured!
✓ Runner "static" installed
✓ Registered ondemand runner!
✓ Initializing Nomad client...
✓ Waypoint runner installed
✓ Runner "static" adopted successfully.
Waypoint server successfully installed and configured!
The CLI has been configured to connect to the server automatically. This
connection information is saved in the CLI context named "install-1662074568".
Use the "waypoint context" CLI to manage CLI contexts.
The server has been configured to advertise the following address for
entrypoint communications. This must be a reachable address for all your
deployments. If this is incorrect, manually set it using the CLI command
"waypoint server config-set".
To launch and authenticate into the Web UI, run:
waypoint ui -authenticate
Advertise Address: 127.0.0.1:9701
Web UI Address: https://127.0.0.1:9702
```
```
$ nomad service list
Service Name Tags
waypoint-server [waypoint]
waypoint-ui [waypoint]
$ nomad service info waypoint-server
Job ID Address Tags Node ID Alloc ID
waypoint-server localhost:9701 [waypoint] 87512324 3345655d
In addition to service discovery support for Nomad, we’ve updated Waypoint’s Nomad task launcher plugin, which launches on-demand runners in Nomad, to report logs in the static runner. This means that short-lived, on-demand runner Nomad jobs, which are purged after the Waypoint job is completed, will stream their logs to the long-lived static runner, whose logs will be consumable long after the on-demand runner job has completed. This is designed to facilitate debugging issues in your builds, deployments, releases, and now pipelines in your on-demand runners. This capability has also been added for Docker and Kubernetes task launcher plugins.
CLI output:
$ waypoint build -local=false
» Building nomad-nodejs-web...
» Operation is queued waiting for job "01GCAA4717TKDPXQETTV41T29A". Waiting for runner assignment...
If you interrupt this command, the job will still run in the background.
Performing operation on "nomad" with runner profile "nomad-task-launcher-test-profile"
» Cloning data from Git
URL: https://github.com/hashicorp/waypoint-examples
Ref: nomad-remote-ops
» Downloading from Git
Git Commit: 88c09d2c1019a21ed4674dd4c7248b740b954767
Timestamp: 2022-09-06 20:07:23 +0000 UTC
Message: Specify runner profile for Nomad.
✓ Running build v8
✓ Building Buildpack with kaniko...
✓ Repository is available and ready: team-nomad/nodejs-example:1
✓ Executing kaniko...
│ Adding label 'io.buildpacks.build.metadata'
│ Adding label 'io.buildpacks.project.metadata'
│ Saving localhost:36615/team-nomad/nodejs-example:1...
│ *** Images (sha256:2cc7a621b53b1ad41ba334024c5ac803a8eb0c02091d2301d00cd6a2d9523
│ eaa):
│ localhost:36615/team-nomad/nodejs-example:1
│ Adding cache layer 'heroku/nodejs-engine:dist'
│ Adding cache layer 'heroku/nodejs-npm:toolbox'
│ INFO[0066] Taking snapshot of full filesystem...
│ INFO[0075] Skipping push to container registry due to --no-push flag
✓ Image pushed to 'team-nomad/nodejs-example:1'
✓ Running push build v7
Created artifact v7
Waypoint on-demand runner (ODR) logs, written to a static runner container in Nomad:
We’ve also improved the Nomad jobspec canary releaser plugin, which was introduced in Waypoint 0.8. It will now no longer be invoked by default with the Nomad jobspec platform plugin, you’ll need to explicitly specify the releaser plugin in your waypoint.hcl file to use it. The resource reporting for the Nomad job, which is promoted by the canary releaser plugin, has also been updated to report the status of the correct Nomad job.
»Interactive waypoint.hcl Generator
Previously, when a first-time user ran waypoint init in an app directory, Waypoint auto-generated a basic waypoint.hcl for the user to then customize to fit their app. Now, in Waypoint 0.10, Waypoint’s CLI can guide users through the process of crafting their first waypoint.hcl file.
Running waypoint init in a new directory now asks the user, “Do you want help generating a waypoint.hcl file? Type 'yes' to initialize the interactive generator or 'no' to generate a template waypoint.hcl file.” Entering “yes” starts the dynamic generator. The generator guides the user through the different built-in plugin options for each stage of the build, deploy, and release stanzas, and provides hints for the user to fill in required configuration.
» Name your app
Please enter an app name: my-first-app
You inputted "my-first-app"
Is this app name correct? (y/N): y
App name confirmed
» Choose build, registry, deployment platform, and releaser plugins
» Configure builder
Select a builder: learn more at https://www.waypointproject.io/plugins. To use a builder that’s not shown here hit return, then edit the .hcl file after it’s been generated.
1: aws-ami
2: aws-ecr-pull
3: docker-pull
4: docker
5: files
6: pack
Please select a plugin by typing its corresponding number or hit "return" to skip this step (1-6): 6
You have selected the pack plugin.
Is this builder plugin correct? (y/N): y
Builder plugin confirmed
You have selected the pack builder plugin.
There are no required fields for this builder plugin, but there may be optional fields you can add to your .hcl file later. See the Waypoint plugin documentation for more information.
Step complete: builder configuration
»Next Steps
The full list of new features and improvements in Waypoint 0.10 can be found in the CHANGELOG. Finally, don’t miss these useful Waypoint resources:
- Download Waypoint 0.10 from the project website.
- Learn more about Waypoint with tutorials on the HashiCorp Learn site.
- Contribute to the Waypoint open source project and participate in its community.
- Extend Waypoint by creating a plugin.
Sign up for the latest HashiCorp news
More blog posts like this one
Terraform, Packer, Nomad, and Waypoint updates help scale ILM at HashiConf 2024
New Infrastructure Lifecycle Management (ILM) offerings from HashiCorp Terraform, Packer, Nomad, and Waypoint help organizations manage their infrastructure at scale with reduced complexity.
HCP Waypoint now GA with enhancements to golden workflow capabilities
Announced last year, HCP Waypoint is now generally available and so are templates and add-ons. The new release features HCP API support and an upgraded workflow for templates.
HCP Waypoint actions is now in public beta
The beta of HCP Waypoint actions to let platform teams expose Day 2+ operations and workflows to developers.