Beta Support for CRDs in the Terraform Provider for Kubernetes

In 2020 we released the kubernetes-alpha project: an experimental provider with the goal of giving practitioners a dynamic way to manage any Kubernetes API resource using HashiCorp Terraform. This new provider contained only a single resource, named kubernetes_manifest, which provided an HCL analog for Kubernetes YAML manifests.

The kubernetes_manifest resource from the kubernetes-alpha project is now in beta, and a part of the official Terraform Provider for Kubernetes. This addition allows users to manage custom resource definitions (CRDs), custom resources, and any additional Kubernetes resources that are not otherwise supported by the official provider — without having to use any additional providers.

»Example Usage

Since the kubernetes_manifest resource is still in beta, we added it to the official provider as an experimental feature that can be enabled in the provider’s configuration block. This resource provides a pure Terraform way of declaring any arbitrary Kubernetes manifest using HCL.

The kubernetes_manifest resource can be used to create any Kubernetes API resource, including custom resource definitions. See the provider documentation page for more information on the additional attributes available for this resource.

»Adoption

We recommend converting your YAML manifests to HCL for use with the provider. You can do this using the yamldecode function in the Terraform console, or with the community tool tfk8s.

You can also use our Learn guide on Managing Kubernetes Custom Resources with Terraform.

»Migration From kubernetes-alpha

If you were already using the kubernetes-alpha provider alongside the official Kubernetes provider, you can simply remove the kubernetes-alpha provider block and provider = kubernetes-alpha attribute and the resource will continue to work with the official provider.

If you were using the kubernetes-alpha provider on its own, you should remove the provider = kubernetes-alpha attribute and replace the provider configuration block with one from the official provider as seen in the example above.

See our migration guide for more information.

»How the kubernetes_manifest Resource Works

Manifests for Kubernetes resources, particularly custom resources, are very dynamic in nature. This means that it’s impractical for us to maintain a provider that is statically built to support the schemas of every possible Kubernetes resource. Just maintaining parity with the core Kubernetes API keeps our team and contributors busy.

As a solution, we created the kubernetes_manifest resource using terraform-plugin-go. This allows us greater access to Terraform’s plugin protocol to generate the type information that’s needed by querying the OpenAPI specification from the Kubernetes API at plan time, and then storing it as an attribute of the type object. Once the plan has been generated, we use server side apply to apply the manifest. This gives us a native Terraform way of expressing a Kubernetes manifest without resorting to strings containing YAML or JSON.

We were then able to merge the code from the kubernetes-alpha repository into the official provider and use terraform-plugin-mux to allow the kubernetes_manifest resource to be used alongside the other Kubernetes resources officially supported by Terraform.

»Limitations

As the kubernetes_manifest resource is dynamic in nature, you need to be aware of a few limitations compared to resources built into the Kubernetes provider.

1. Access to the Kubernetes API is required at plan time. This means that a Kubernetes cluster must be created and available in a separate Terraform apply operation before you can apply a kubernetes_manifest resource. This is because the provider has to fetch the OpenAPI specification from the Kubernetes API to generate the type information required by Terraform.

2. Custom resource definitions must be applied before custom resources. As above, this is because the provider queries the Kubernetes API for the OpenAPI specification for the resource supplied in the manifest attribute. If the CRD doesn’t exist in the OpenAPI specification during plan time then Terraform can’t use it to create custom resources.

3. Each manifest resource can contain only one Kubernetes resource. Each Terraform resource represents the state for a single Kubernetes resource, so your manifest can not contain multiple YAML documents or a manifest that is a list of resources – they must be broken up into individual resources.

4. Slower performance. When using the kubernetes_manifest resource, the provider has to query for and generate the type information for that resource’s schema. So we recommend that if your resource is already supported as a named resource in the provider, you should use that, as the plan will run faster.

»Future Work

We added the kubernetes_manifest feature to the provider as an opt-in experiment, so we expect to collect feedback on how the resource fits into existing workflows that will shape any changes before we remove the experiment flag and consider the feature generally available. Please file issues generously if you encounter any.

We also expect to add import support, greater flexibility in waiting for conditions and events, and a data source that is an analog to the manifest resource.

»Want to Contribute?

The kubernetes-alpha repository has now been archived and the provider code and issues have been moved to the repository for the official provider. Please see our contributing guide and talk to us on Slack (#terraform-providers on the Kubernetes Slack) if you want to help improve the Terraform and Kubernetes experience.

Special thanks to Alex Somesan and Stef Forrester on the Terraform OSS team for all their hard work developing and testing the provider, and to all of the contributors to the kubernetes-alpha repository for making the kubernetes_manifest feature possible.