Terraform, Packer, Nomad, and Waypoint updates help scale ILM at HashiConf 2024

Today at HashiConf in Boston, we are pleased to announce our latest capabilities across our Infrastructure Lifecycle Management (ILM) portfolio, including HashiCorp Terraform, Packer, Nomad, and Waypoint, to help customers build, deploy, and manage infrastructure at scale.

Our latest ILM capabilities help organizations manage infrastructure across Day 0, 1, and 2+.:

• Day 0
  • HCP Packer CI/CD pipeline metadata (GA) to track critical CI/CD information in build pipelines through integrations with GitHub and GitLab
  • HCP Packer bucket-level RBAC (GA) to gain further control over image permissions management
• Day 1
  • HCP Terraform Stacks (public beta) to simplify infrastructure management at scale
• Day 2+
  • HCP Terraform module lifecycle management (public beta) to reduce the overhead of module management
  • Terraform migrate (public beta) to accelerate migration from the community edition to HCP Terraform and Terraform Enterprise
  • HCP Waypoint (GA) with templates (GA) and add-ons (GA), now with API support and an upgrade workflow for templates
  • Nomad enhanced GPU support (GA)

This blog looks at how each of these new features contribute to speeding, securing, and simplifying the full lifecycle management of infrastructure.

»Day 0: Build securely with infrastructure as code

As organizations plan and define the requirements of their services, Day 0 is the time to lay a strong foundation for ILM. Organizations need a programmatic approach to defining and provisioning application environments quickly and securely. They must prevent vulnerabilities in the software supply chain, including their base images and build artifacts, and ensure users have appropriate access based on their roles to mitigate security risks.

»HCP Packer improves metadata visibility and access control

HCP Packer recently added CI/CD pipeline metadata views that give users even more visibility into artifact creation by letting them track critical CI/CD information such as pipeline IDs, job names, details on the operating system, VCS commits, and more. This addition grants HCP Packer level 1 SLSA compliance by providing a basic level of source code identification that can help organizations make risk-based security decisions. With this visibility, organizations can address risks earlier in the infrastructure deployment process.

Another key addition to HCP Packer is bucket-level RBAC, which helps admins define user access at the bucket level. This increased access granularity lets developers create buckets within the same project that they can access, while still being walled off from full-project access when they don’t need it. Specific permission can be assigned at the bucket level for actions such as creating, updating, and deleting artifact versions and more. With this improvement, organizations can now ensure sensitive golden images remain protected from unauthorized modifications while giving developers the self-service capabilities they need to be agile and efficient.

»Day 1: Deploy infrastructure at scale without complexity

On Day 1, when developers are ready to provision the infrastructure needed to deploy an application, they want to scale quickly to meet business needs without complexity. They don’t want to waste valuable time repeating complex workarounds or repetitive manual processes.

»HCP Terraform Stacks provide a built-in way to scale

Last October, we announced the private preview of HCP Terraform Stacks, a new way to simplify infrastructure provisioning and management at scale, reducing the time and overhead of managing infrastructure. Stacks empower users to rapidly create and modify consistent infrastructure setups with differing inputs, all with one simple action. Stacks also eliminate the need to manually track and manage cross-configuration dependencies as multiple Terraform modules can be organized and deployed together in a Stack.

Today, we’re excited to announce the public beta of Terraform Stacks for all new HCP Terraform plans based on resources under management (RUM). During the public beta, HCP Terraform users can experiment with Stacks to provision and manage up to 500 resources for free, including a new Kubernetes use case and two new features: deferred changes and orchestration rules. Go to HashiCorp Developer to learn how to create a Stack in HCP Terraform.

The new Kubernetes use case streamlines the provisioning and management of Kubernetes workloads by allowing customers to deploy Kubernetes in one single configuration instead of managing multiple, independent Terraform configurations. We see Kubernetes deployments that often have this challenge where there are too many unknown variables to properly complete a plan. With Stacks, customers can drive a faster time-to-market with Kubernetes deployments at scale without going through a layered approach that is hard to complete within Terraform.

The reason we can enable the Kubernetes use case hinges on a new feature: deferred changes. This feature allows Terraform to produce a partial plan when it encounters too many unknown values — without halting operations. This helps users work through unknown-value situations more easily, accelerating the deployment of certain workloads with Terraform, most notably Kubernetes.

Orchestration rules, defined in HCL, allow customers to automate repetitive actions. For example, at the launch of the public beta, users can auto-approve a plan when certain orchestration checks and criteria are met. This simplifies the management of large numbers of deployments by codifying orchestration checks that are aware of plan context in the Terraform workflow.

To learn more, read our updated blog Terraform Stacks, explained, refer to our Stacks documentation, and get hands-on experience in our Terraform Stacks tutorials.

*Stacks orchestration rules will be available to all HCP Terraform RUM plans during public beta

»Day 2+: Manage and optimize infrastructure operations continuously

After deployment, on Day 2 and beyond, organizations need to manage their environments and optimize their operations continuously. End-of-life clean-up is a key part of that story, whether it's for Terraform workspaces or modules. Ephemeral workspaces have continued to advance their utility for resource clean-up, with the recent project-scoped auto-destroy setting enhancement. Terraform also provides excellent tools for managing the creation and organization of golden modules, but visibility and end-of-life operations are also important to consider here as well. And when teams are ready to scale golden patterns and workflows, they’ll need tools that help them build an internal developer platform (IDP) to make infrastructure easily accessible for developers at any skill level.

»HCP Terraform module lifecycle management provides comprehensive module visibility and controls

The HCP Terraform private registry makes it easy to publish and discover modules internally, but doesn’t fully address the end-of-life states of the module lifecycle. As a consequence, deprecating outdated versions and controlling their distribution could become difficult, especially with large quantities.

From a management perspective, without a native workflow to provide visibility, like usage reports, and communication mechanisms with the right module consumers, organizations struggle to know how much any particular module version is being used, and who to ask for upgrades. From a security and compliance perspective, without a proper way to signal the deprecation of outdated modules, organizations are at risk of using obsolete and out-of-compliance configurations.

Today, we are introducing module lifecycle management improvements in public beta to provide a systematic way to provide visibility, improve communication, and gain control throughout the module lifecycle. These improvements will simplify the complexity of module version management and reduce its overhead while also reducing security and compliance risks.

To take advantage of new module lifecycle management features, the platform team can use change requests in the HCP Terraform explorer, to communicate infrastructure lifecycle events such as:

• Module deprecation
• Drift remediation
• Provider upgrades
• Infrastructure changes

For example, platform teams can use module deprecation in the private registry to provide customized warnings about outdated module versions without interruption.

Combined with team notifications in Terraform’s teams settings, which help configure a destination per-team communication channel, the requests always get to the right owners proactively in addition to showing up in the HCP Terraform workspace UI. Then users can use saved views in HCP Terraform explorer to track the progress of change requests for follow-up.

To learn more about each feature, refer to our documentation on module deprecation, change requests, and team notifications. Change requests, team notifications, and module deprecation are only available in the HCP Terraform Plus tier. Saved views are available for all Terraform plans based on RUM.

»Terraform migrate accelerates migration from the community edition

Some customers are interested in trying out HCP Terraform or Terraform Enterprise, but find the migration process from Terraform Community Edition manual, time-consuming, and daunting. This slows the time-to-value for teams that want to migrate and causes friction for organizations adopting a commercial edition of Terraform.

To help simplify and accelerate migrations from Terraform Community Edition to HCP Terraform or Terraform Enterprise, we’ve released Terraform migrate in public beta. Terraform migrate automates the tedious process of migrating workflows at scale in a way that is aligned with our best practices: HashiCorp Validated Designs. The Terraform migrate utility also reduces the risks of mistakes with a consistent migration process. All actions are previewed before changes are made, and Terraform migrate ultimately reduces the total cost of ownership by reducing the time spent performing manual migrations. To learn how Terraform migrate works, please refer to our documentation for Terraform migrate.

»HCP Waypoint is now generally available to help provide self-service infrastructure to developers

As organizations quickly grow their infrastructure footprint across cloud environments, it can result in an overwhelming increase in scope and complexity in a short period of time. Enterprises can have thousands of downstream developers who need infrastructure to build applications, many of which are not well-versed in the specifics of infrastructure configuration. To scale effectively, organizations can set up an internal developer platform (IDP) that gives platform teams a central control point through which they can provide golden infrastructure workflows. This gives developers an easy way to consume these patterns in a self-service fashion.

HCP Waypoint, a product for creating an IDP to make infrastructure easily accessible, is now generally available. With this release comes templates for provisioning underlying infrastructure and add-ons to manage application dependencies. These components are also GA as of today.

The GA release of HCP Waypoint also includes a new upgrade workflow that pushes updates to Waypoint applications when the Waypoint template is updated, including updates to the underlying Terraform module version. We also support using the HCP API to access Waypoint resources such as templates, add-ons, and applications.

We’re also releasing variable support for actions (currently in public beta) to allow platform teams to specify input variables and their values when creating actions.

HCP Waypoint is now available to all HCP Terraform Plus users. Variable support for actions will be available to HCP Terraform RUM Plus plans during public beta. To learn more, refer to our Waypoint product page and see our blog post: HCP Waypoint now GA with enhancements to golden workflow capabilities.

»Nomad adds support for Multi-Instance GPU (MIG), quotas for device resources, golden job versions, and more

Today we’re announcing Nomad 1.9, which now has the ability to schedule workloads onto an NVIDIA Multi-Instance GPU (A100 and H100 GPUs). As GPUs become more critical in high-performance computing tasks such as machine learning and generative AI, Nomad’s ability to schedule GPU workloads has continued to evolve and improve alongside the technology. Nomad’s MIG support now allows operators to partition GPU resources across multiple users for optimal GPU utilization. In addition, we now provide the ability to assign quotas to GPUs and GPU instances to help restrict aggregate usage of resources by namespace or region.

Nomad 1.9 also brings NUMA awareness and quotas for device resources, improving Nomad’s device orchestration capabilities. In addition, Nomad also introduces golden job versions, which allow operators to tag and track their Nomad jobs for reuse. By reusing the “golden” jobs that follow organization best practices, orchestration with Nomad becomes more efficient and secure.

To learn more, see our blog post: Nomad 1.9 adds NVIDIA MIG support, golden job versions, and more.

»Get started today

With thousands of customers, our Infrastructure Lifecycle Management portfolio, including HashiCorp Terraform, Packer, Nomad, and Waypoint, offers a blueprint to cloud success as organizations are rethinking their cloud programs. For some organizations who have struggled with the transition to cloud, it’s a second chance to do cloud right.

You can try many of these new features now and customers on the HashiCloud Cloud Platform can get them applied automatically with no disruption to existing workflows. HCP customers can also begin using the integrated product workflows that combine our ILM products with solutions from our Security Lifecycle Management (SLM) portfolio to simplify common use cases like image management within infrastructure provisioning and privileged access management.

If you are new to our ILM products, you can get started in minutes using the HashiCorp Cloud Platform or sign up for HCP Terraform, HCP Packer and HCP Waypoint to get started for free today. To learn more about Nomad, check out our tutorials.

If you'd like to see a deep dive webinar recap of these announcements, sign up for our ILM HashiConf recap.