Making Terraform Provider Development More Accessible

Since the first provider was released in 2014, over 1,900 HashiCorp Terraform providers have been written. Terraform providers enable everything from standing up critical infrastructure to ordering a pizza, and all of this is written on top of the framework we defined six years ago.

Terraform has changed a lot in the last six years, however. Not only has our ecosystem grown vastly more extensive and diverse, but our understanding of Terraform has evolved and been refined over time, too. We understand the problem space a lot more than we did six years ago, and our understanding of the constraints and boundaries of provider development has evolved. Terraform 0.12 is probably the most visible effect of this shift, formalizing and codifying many of the blurry edges that long existed in our framework, and exposing new possibilities and integration points for developers.

Provider developers haven’t been able to take advantage of many of these new features and integration points yet. The framework that has enabled Terraform’s growth wasn’t built with them in mind, and the large ecosystem that makes Terraform so useful makes modifications to the framework costly. Our providers have thousands of resources written against this framework, meaning backward-incompatible changes to the framework are largely impossible. Even smaller upgrades designed to be minimally invasive, like v2.0.0 of the SDK, are expensive for our ecosystem. And yet, we need to be able to deliver new features and functionality to provider developers, and that sometimes requires us to change interfaces. Layering new features and functionality on the existing framework has made it harder to get started with Terraform provider development, with a lot of context needed before progress is visible.

The Terraform Plugin SDK team has spent much of 2020 working on this problem, and we’re excited to share the first steps towards a solution with the community today.

»Announcing New Go Libraries for the Terraform Plugin Protocol

First, we’re excited to announce the availability of the terraform-plugin-go module, available starting today. This module is a light wrapper around the Terraform protocol, with no behavior or abstractions of its own. Its goal is to enable advanced Terraform provider developers to have access to the raw protocol, giving them the ability to take advantage of all of Terraform’s functionality, at the expense of fewer guardrails, less guidance, and more verbosity. This is not meant as a replacement for the framework that exists today; it is a lower level of abstraction and will prove challenging for beginning Terraform developers to use correctly.

Secondly, we’re pleased to also be releasing terraform-plugin-mux, a Go module to allow multiplexing terraform-plugin-go servers together. This module allows provider developers to have several different provider implementations exposed as a single Terraform provider server, and it will route Terraform’s RPC requests to whichever implementation supports the resource the request is for. This allows provider developers to upgrade between provider frameworks one resource at a time, instead of requiring them to migrate all resources at once, as long as the frameworks are built on terraform-plugin-go.

»Terraform Plugin SDK Updates

Finally, we’re excited to share that the Terraform Plugin SDK has been migrated onto terraform-plugin-go as of version 2.2.0, released today, which should be a transparent upgrade for provider developers with no observable changes. This will not bring any new features or capabilities to the SDK. It does, however, allow provider developers to use terraform-plugin-go and the Terraform Plugin SDK in the same provider, transparently, using terraform-plugin-mux.

»What’s Next?

We’re very excited about these new modules and believe they enable a promising future for Terraform provider development. The introduction of terraform-plugin-mux will enable us to more aggressively develop enhancements to the Terraform provider developer experience for new resources, without breaking any of the existing resources that provider developers have spent so much time writing. The introduction of terraform-plugin-go will enable developers to take full advantage of the formalized and expanded capabilities of Terraform that were introduced in Terraform 0.12. And the migration of the Terraform Plugin SDK to use the terraform-plugin-go types enables developers maintaining existing providers to take advantage of this functionality today, without needing to modify their existing resources at all.

We are still developing these new modules. They are, today, released as version 0.1.0. This denotes that they are usable, and we are confident they work, but we need some help from the community to ensure we’re offering the right interfaces. Developers adopting these modules should do so with an intention to keep an eye on the CHANGELOG and with the understanding that interfaces may change in backward-incompatible ways while we grow more confident in the design of the packages.

For more information, including demos and an exploration of the new capabilities these releases unlock, be sure to tune in to our HashiCorp Live event on November 20th at 1:00 p.m. Pacific, where we’ll be talking about both these packages. In the meantime, feel free to ask questions and let us know what you think in the forums. Happy Terraforming!