Deploy Any Resource with the Kubernetes Provider for Terraform

This provider is now out of alpha and you can now read a consistenly updated tutorial on how to use it: Manage Kubernetes Custom Resources.

We are pleased to announce the alpha release of a new version of the Kubernetes Provider for HashiCorp Terraform. The kubernetes-alpha provider lets you package, deploy, and manage all Kubernetes resources, including Custom Resource Definitions, using HashiCorp Configuration Language (HCL). With the addition of the kubernetes-alpha provider, you can now manage the full lifecycle of Kubernetes and its workloads using Terraform. All Kubernetes resources are accessible through this new provider.

In addition to complete coverage of current Kubernetes resources, the provider uses a Kubernetes feature called Server-side Apply (SSA). SSA is a new merging algorithm introduced in Kubernetes 1.16. It will become the default interaction model in a future version of Kubernetes. With SSA enabled, the provider defers to the Kubernetes API to detect differences in attributes and handle conflict detection for Terraform plan and apply. SSA attributes are now assigned a manager, and the API tracks the clients that changed that attribute. A journal of changes is kept in the resource metadata so users can easily determine which attributes have been modified by Terraform as the manager. By leveraging SSA, Terraform delivers a user experience closer to what Kubernetes users are familiar with and expect from native tools such as kubectl.

»Usage

This provider is experimental and you cannot install it from the Terraform provider registry for now. To try out this provider, you will need to download the latest release from the github repository. Please download both the source code as well as the binary appropriate to your architecture.

We will assume that there is an existing installation of Go and that the $GOPATH and $GOBIN variables are appropriately populated. We also assume that you have a Kubernetes test environment available to use. Please ensure that the Kubernetes cluster you choose meets the minimum version requirements (1.17 or newer). This example will work with clusters created via minikube or kind.

Install your provider binary by copying it into your Terraform plugins folder.

$ cp terraform-provider-kubernetes-alpha ~/.terraform.d/plugins/terraform-provider-kubernetes-alpha

In this post, we’ll walk through the usage of one of several examples that are available in the repository. This example will install a Custom Resource Definition (CRD) using the kubernetes_manifest resource available in the new provider. To try this example out, change directories into the “examples/crd” folder of the kubernetes-alpha source code that you downloaded earlier. We will also check that the latest version of Terraform is installed.

$ cd examples/crd
$ terraform version

In the crd folder, you should see a single file - main.tf that has the contents below:

provider "kubernetes-alpha" {
 server_side_planning = true
}
 
resource "kubernetes_manifest" "test-crd" {
 provider = kubernetes-alpha
 
 manifest = {
   apiVersion = "apiextensions.k8s.io/v1"
   kind = "CustomResourceDefinition"
   metadata = {
     name = "testcrds.hashicorp.com"
     labels = {
       app = "test"
     }
   }
   spec = {
     group = "hashicorp.com"
     names = {
       kind = "TestCrd"
       plural = "testcrds"
     }
     scope = "Namespaced"
     versions = [{
       name = "v1"
       served = true
       storage = true
       schema = {
         openAPIV3Schema = {
           type = "object"
           properties = {
             data = {
               type = "string"
             }
             refs = {
               type = "number"
             }
           }
         }
       }
     }]
   }
 }
}

Notice that the kubernetes_manifest resource contains a provider attribute and a manifest attribute, which is an HCL representation of your YAML manifest. In the next section, we describe how you can ease the conversion process of your YAML manifests into the HCL input required by this resource.

In order to demonstrate the provider behavior, we will assume that you already have a Kubernetes test environment set up and that your kubeconfig is configured to use that environment. You will apply the example CRD to your Kubernetes cluster using Terraform and ensure that Terraform is initialized with the alpha provider.

First, initialize Terraform and check to see what CRDs are currently installed.

$ terraform init
$ kubectl get crds

Then, plan and apply the Terraform configuration in the main.tf file to create a test CRD in the cluster. A quick check with kubectl shows that the CRD was successfully applied.

$ terraform plan -out=crds.tfplan
$ terraform apply crds.tfplan
$ kubectl get crds

Finally, clean up by destroying the CRD via Terraform. This is followed by a final check to ensure the CRD was correctly removed.

$ terraform destroy
$ kubectl get crds

This screencast demonstrates the steps described and the resulting outputs.

As this and the other examples in the kubernetes-alpha repository demonstrate, the new kubernetes-alpha provider can deploy any resource to your Kubernetes cluster via a conversion of the YAML manifest into an HCL representation. We are excited about the possibilities this provider brings and we’re looking forward to hearing your feedback.

»Converting YAML to HCL

We recommend that you convert your YAML manifests into static HCL to take advantage of Terraform’s ability to interpolate values into the resource’s attributes. There are a couple of options available to you to automate this conversion.

1. Terraform built-in function

You can use the built-in yamldecode() function available in Terraform with the path to your YAML manifest file to do a one-time conversion to HCL.

$ echo 'yamldecode(file("my-manifest-file.yaml"))' | terraform console

Note that this method requires that you only have one YAML document per manifest.

2. External utility

Alternatively, you could use an experimental utility that allows you to pass your YAML manifest as a command line input and produces the resulting HCL. For YAML manifests containing multiple documents, we recommend creating unique kubernetes_manifest resources in your Terraform configuration for each document.

In general, we recommend using this procedure as a one-time conversion step followed by management of your HCL resources directly in Terraform.

»Use Cases

• Dry-run changes to Kubernetes resources and evaluate them with Terraform plans
• Deploy Kubernetes Custom Resources and Custom Resource Definitions with Terraform
• Package and distribute multiple Kubernetes resources, such as Kubernetes Operators, as Terraform Modules

»Current Limitations

This provider is experimental. It relies on new features introduced in Terraform v0.12, such as the rich type system for resource attributes. As we employ these new features in Terraform, we are looking to build the best possible user experience in the Kubernetes provider. However, this is our first such endeavor and we would like users to be aware that it is still a moving target. We ask that you try this provider out and give us feedback, but keep in mind that this provider is not yet ready for production workloads. At this stage, complete rewrites of the provider implementation and user experience are still within scope.

Listed below are some details on known limitations of this provider that we are currently investigating.

• Supports only Kubernetes 1.17 and above
  • The provider makes use of server-side apply features introduced in Kubernetes 1.16 but unstable in that version. The minimum recommended version is 1.17.
• Some limitations when using local planning
  • Updates may not work as expected if they produce additional resource attributes not present at the time of resource creation.
• Some limitations when using server-side planning
  • Cannot be used when a Kubernetes resource is sourcing any of its attribute values from another Kubernetes resource that was not present in the cluster prior to the current operation.

»Conclusion

For more information on the new Kubernetes Provider, see the Github repository. To discover other ways to manage Kubernetes with Terraform, review our guides on HashiCorp Learn.

We would love to hear your feedback and expand on this project! Post bugs, and feature requests regarding the kubernetes-alpha provider by opening an issue at hashicorp/terraform-provider-kubernetes-alpha!