Products & Technology

Predictable plugin loading in Packer 1.11

To boost stability, Packer 1.11 introduces a predictable plugin loading approach, loading only binaries from its plugin directory with accompanying SHA256SUM files.

Mar 12 2024Wilken Rivera, Lucas Bajolet

In HashiCorp Packer 1.11, now available as an alpha release, we are introducing a predictable approach to plugin loading. With this change, Packer will no longer load plugin binaries installed outside of its plugin directory, nor will Packer load plugin binaries without their respective SHA256SUM file. To aid this transition, Packer 1.11 includes tooling updates to simplify the plugin loading process.

»What's changing in Packer 1.11?

The Packer team has been consistently working on reducing the pain points around plugin usage introduced by the required plugins in Packer 1.7. In user interviews, community forum posts, and various GitHub issues, we continue to hear about difficulties installing plugins into the correct directories and plugin developers facing challenges with the tooling to test their locally built binaries due to the various plugin loading options in Packer.

As it stands today, plugins following one of the two naming conventions, packer-plugin-happycloud or packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64, placed within one of the known directories will get automatically loaded by Packer. The discovery and loading of a plugin within a known directory is a feature that has been in place since the early days of Packer. But in Packer 1.7, with the introduction of required_plugins and packer init, the flexibility of automatic plugin discovery could be confusing to some.

»Local plugin installation tools

In Packer 1.10 we introduced the ability to install a locally sourced plugin using packer plugins install --path, which made the use of locally installed plugins compatible with required_plugins and packer init. In Packer 1.10.2, we extended this behavior to support development plugin binaries — binaries that report "dev" as part of their plugin version.

The Packer team decided on "dev" prereleases over "alpha", "beta", and "rc" for plugins to minimize the level of complexity around version pinning used for required_plugins and packer init.

»Using development binaries with Packer

Plugins required through required_plugins have a version constraint that dictates the version of a plugin needed for executing a build. These constraints do not include development versions because installation through commands like packer init is unsupported.

However, development plugins are now evaluated at runtime provided their version matches the constraints specified. For example:

amazon = {
  source = "github.com/hashicorp/amazon"
  version = ">= 1.1.0"
}

Given the specified version constraint above, only versions greater than or equal to 1.1.0 will be considered.

If you have a development binary (i.e. a manually built plugin binary) installed, Packer will use it if:

It is the highest compatible version installed.
There is no final plugin version with the same version number installed alongside it.

So assuming the following hierarchy:

/Users/dev/.packer.d/plugins
└─ github.com
   └─ hashicorp
    	└── amazon
          ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64
          ├── packer-plugin-amazon_v1.1.0_x5.0_darwin_arm64_SHA256SUM
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64
          └── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM

Version 1.1.1-dev of the Amazon plugin will match the specified version constraint and be used for executing the Packer build.

If, however, a 1.1.1 release version of the plugin is available, it will take precedence over the development binary:

/Users/dev/.packer.d/plugins
└─ github.com
   └─ hashicorp
    	└── amazon
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64
          ├── packer-plugin-amazon_v1.1.1-dev_x5.0_darwin_arm64_SHA256SUM
          ├── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64
          └── packer-plugin-amazon_v1.1.1_x5.0_darwin_arm64_SHA256SUM

Here, version 1.1.1 of the plugin will match the specified version constraint and be used for executing the Packer build.

»Dropping support for legacy single-component plugins

In Packer 1.11.0, we removed Packer's ability to load single-component plugins. These are legacy plugins following the previously deprecated naming convention of packer-builder-happycloud or packer-provisioner-happycloud-shell in favor of supporting only multi-component Packer plugins like the Docker plugin for Packer.

»Stricter plugin loading

In Packer 1.11, we are dropping support for loading plugin binaries following the naming convention of packer-plugin-name. Packer will now load only plugins stored under PACKER_PLUGIN_PATH using the expected namespaced directory and CHECKSUM files. This change drops support for loading plugin binaries in Packer's executable directory or a template's current working directory:

/Users/dev/.packer.d/plugins
└── github.com
    └── hashicorp
        └── happycloud
            ├── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64
            └── packer-plugin-happycloud_v0.0.1_x5.0_darwin_arm64_SHA256SUM

»What does this mean for Packer users?

As Packer users, if your templates leverage the required_plugins Packer block and you're installing plugins via packer init, your workflows will continue working as they do today. If, however, you use plugins that live outside of Packer's known plugin directory or manually manage Packer plugin directories, you may need to change your plugin management workflow.

Packer will no longer support the loading of plugin binaries installed alongside the Packer binary or in the current working directory. Instead of manually placing a downloaded binary into the executable or current working directory, we encourage you to run the command packer plugins install –path <path-to-downloaded-extracted-binary> github.com/hashicorp/happycloud to install the binary into a Packer compatible path.

Running the install command with the --path option will generate the associated SHA256SUM file for validating the locally installed plugin. If you prefer to manage the installation manually, you can do so but you will be required to manually construct the namespaced sub-directories and SHA256SUM file.

This may sound like a lot of work for installing a binary, but it provides a consistent manner for installing plugins, ensures Packer will load the correct binary at runtime, and makes all plugins compatible with required_plugins and packer init.

HCL (HashiCorp Configuration Language) users can safely pin plugin versions and use dev prereleases without having to change any template, as the use of development binaries with required_plugins now works out the box.

»Installing development binaries in Packer

The changes mentioned in this blog post give plugin developers and users who have to build their own versions of trusted plugins the ability to use these binaries without conflicting with the Packer plugin pinning mechanism. Practitioners’ use of manually built plugin binaries, what HashiCorp calls “development binaries”, is a common practice given the open source nature of Packer plugins. In Packer 1.11, we've updated the plugin tooling to treat development binaries as first-class citizens in Packer.

A full explanation of how to build development binaries has been documented within the Packer plugin scaffolding repository. Below is a general overview of the new workflow for using development binaries.

As an example, to build a custom version of the Docker plugin and install it so Packer will be able to use it, you may follow these steps:

Clone the plugin's GitHub repository.
In the plugin directory root, run go build to build the plugin as a development binary.
Use the packer plugins install command to install the development binary.
Run a Packer build with the newly installed plugin.

~> git clone https://github.com/hashicorp/packer-plugin-docker.git
~> cd packer-plugin-docker
 
~> go build -ldflags="-X github.com/hashicorp/packer-plugin-docker/version.VersionPrerelease=dev" -o packer-plugin-docker-dev
 
# Lets validate its a development prerelease
~> ./packer-plugin-docker-dev describe
{"version":"1.0.10-dev","sdk_version":"0.5.2","api_version":"x5.0","builders":["-packer-default-plugin-name-"],"post_processors":["import","push","save","tag"],"provisioners":[],"datasources":[]}
 
# Lets install the development binary
~> packer plugins install --path packer-plugin-docker-dev github.com/hashicorp/docker
Successfully installed plugin github.com/hashicorp/docker from $HOME/Development/packer-plugin-docker/packer-plugin-docker-dev to ~/github.com/hashicorp/docker/packer-plugin-docker_v1.0.10-dev_x5.0_darwin_arm64

Note: For convenience, the Makefile within the Packer plugin scaffolding repository has been updated to automate the installation of building and installing development binaries via make dev.

»Next steps

We invite you to test the Packer 1.11 alpha release, which is available on HashiCorp Releases. Please let us know how the new plugin loading experience works for you.

If you encounter any issues or have any suggestions on how we can improve the loading experience, feel free to start a discussion on the Packer GitHub issue tracker or community forum.

Predictable plugin loading in Packer 1.11

»What's changing in Packer 1.11?

»Local plugin installation tools

»Using development binaries with Packer

»Dropping support for legacy single-component plugins

»Stricter plugin loading

»What does this mean for Packer users?

»Installing development binaries in Packer

»Next steps

Sign up for the latest HashiCorp news

More blog posts like this one

New SLM offerings for Vault, Boundary, and Consul at HashiConf 2024 make security easier

Terraform, Packer, Nomad, and Waypoint updates help scale ILM at HashiConf 2024

HCP Packer now tracks CI/CD pipeline metadata