Products & Technology

Terraform Cloud Operator v2 for Kubernetes is now GA

Terraform Cloud Operator v2 for Kubernetes is now generally available, enhancing resource management and scaling of Terraform Cloud agents.

Nov 07 2023John Houston, Vishnu Ravindra

Initially launched as v1 in February 2021, the Terraform Cloud Operator for Kubernetes enabled users to manage Terraform Cloud workspaces through a single Kubernetes custom resource. Using the Operator brings Terraform Cloud’s proper state handling and locking, sequential execution of runs, and established patterns for injecting secrets and provisioning resources into your Kubernetes-native workflows.

We’ve been working on a v2 iteration of the Operator that now uses multiple custom resources and today, we’re making Terraform Cloud Operator v2 generally available. Note that the Operator works with both Terraform Cloud and Terraform Enterprise. This post reviews the details and benefits of the enhanced v2.

»Terraform Cloud Operator v2 additions

The v2 release of the Terraform Cloud Operator focuses on three main areas to remove scaling limitations around Kubernetes-Terraform Cloud workflows:

»Flexible resource management

Instead of a single Workspace custom resource that managed both workspaces and API-driven Terraform runs, the Operator now features multiple custom resources, each with separate controllers for different Terraform Cloud resources. The Terraform Cloud Operator v2 includes three custom resource definitions to help manage Terraform Cloud resources:

AgentPool manages Terraform Cloud agent pools with auto-scaling support
Workspace manages Terraform Cloud workspaces
Module implements API-driven run workflows to provision infrastructure

Splitting the v1 Workspace into separate Workspace and Module resources in the second version of the Operator was the result of user feedback, which led us to conclude that a clearer distinction between the entities will make it easier to deploy and manage workspaces at scale. For example, using the separate Module resource, users can execute API-based runs in workspaces that are not managed by the Operator. Similarly, user feedback led to the addition of the AgentPool resource to simplify the deployment of flexible agent pools within Kubernetes clusters.

»Namespace management

The introduction of the --namespace option allows users to tailor the Operator’s watch scope to specific namespaces, enabling more fine-grained resource management as setups grow.

»Configurable synchronization

With the new --sync-period, synchronization between custom resources and Terraform Cloud is faster than ever, ensuring timely updates and smoother operations.

»Observability

Metrics for each controller are exposed in standard Prometheus format to provide administrators with a familiar way to collect usage and performance data from the Operator. The v2 Operator also produces more detailed logging and controllers produce event messages for each custom resource.

»Workspace and module custom resource examples

Workspaces in Terraform Cloud contain all necessary components for Terraform to manage distinct sets of infrastructure. Modules act as reusable packages of multiple resources, utilized collectively.

The Workspace resource allows users to create Terraform Cloud workspaces and manage many settings including execution mode, team permissions, variable values, version control settings, notifications, and more. Below is a sample Workspace resource to create an example-workspace with remote execution mode.

---
apiVersion: app.terraform.io/v1alpha2
kind: Workspace
metadata:
  name: example-workspace
spec:
  organization: example-org
  token:
    secretKeyRef:
      name: tfc-token
      key: token
  name: example-workspace
  description: Example workspace
  applyMethod: auto

Now you can also create a Module resource to trigger an API-driven run within that workspace by sourcing a Terraform module from the public or private registry:

---
apiVersion: app.terraform.io/v1alpha2
kind: Module
metadata:
  name: example-module
spec:
  organization: example-org
  token:
    secretKeyRef:
      name: tfc-token
      key: token
  destroyOnDeletion: true
  module:
    source: "example-org/examples/example"
    version: "1.2.3"
  workspace:
    name: example-workspace

»Terraform Cloud agents custom resource examples

The other custom resource that was added to the Terraform Cloud Operator v2 — the AgentPool resource — allows you to create, configure, and scale Terraform Cloud agent pools based on demand. Terraform Cloud agents enable communication between Terraform Cloud and isolated, private networks or on-premises infrastructure. Once set up, these agent pools can be linked to your workspaces based on your specific needs.

Below is a sample AgentPool resource. This will create an agent pool, create a Kubernetes deployment for the agents, and automatically scale the number of replicas of that deployment to match the queue of runs that are waiting for an agent.

---
apiVersion: app.terraform.io/v1alpha2
kind: AgentPool
metadata:
  name: example-agent-pool
spec:
  organization: example-org
  token:
    secretKeyRef:
      name: tfc-token
      key: token
  name: example-agent-pool
  agentTokens:
  - name: example-token
  agentDeployment:
    spec:
      containers:
      - name: tfc-agent
        image: "hashicorp/tfc-agent:1.13.1"
  autoscaling:
    minReplicas: 1
    maxReplicas: 3
    cooldownPeriodSeconds: 60
    targetWorkspaces:
    - wildcardName: example-*

You can then configure a Workspace resource to use this agent pool:

---
apiVersion: app.terraform.io/v1alpha2
kind: Workspace
metadata:
  name: example-workspace
spec:
  organization: example-org
  token:
    secretKeyRef:
      name: tfc-token
      key: token
  name: example-workspace
  description: Example workspace
  applyMethod: auto
  executionMode: agent
  agentPool:
    name: example-agent-pool
  tags:
  - example

»Key high-level benefits

The latest version of the Terraform Cloud Operator streamlines infrastructure management, allowing platform teams to offer a Kubernetes-native experience for their users while standardizing Terraform workflows. It simplifies the management of Terraform Cloud workspaces and agent pools, ensuring efficiency and consistency across operations. Here are three high-level benefits of this new release:

Increased agility: Platform teams can now provide application developers with Kubernetes-native workflows, while ensuring the use of approved Terraform modules. They also now benefit from auto-scaling agent pools for a cost-effective and reliable Terraform Cloud execution environment.
Reduced risk: Platform and security teams can now retain visibility and policy controls over infrastructure provisioned with Terraform Cloud, ensuring security and compliance. This allows platform and security teams to manage and mitigate risks effectively.
Cost efficiency: The Terraform Cloud Operator provides Kubernetes-focused organizations with a streamlined method for infrastructure management. Organizations can now leverage the cost benefits of platform teams reduction of time spent developing and supporting custom tooling.

»Get started on Terraform Cloud and Terraform Cloud Operator

For additional differences between v1 and v2, read our FAQ on the Terraform Cloud Operator’s GitHub repo.

Read Terraform Cloud Operator for Kubernetes overview to get started with the Operator. Learn more by reading our tutorial on how to set up the new Operator: Deploy infrastructure with the Terraform Cloud Kubernetes Operator v2. If you’re upgrading to v2 of the Terraform Cloud Operator from v1, please check out the migration guide.

Please share any bugs or enhancement requests with us via GitHub issues or join the conversation in the Terraform discussion forum. As always, we look forward to your feedback. This Operator couldn’t have come this far without strong community interest and thoughtful feedback. You have our sincerest thanks!

If you are completely new to Terraform, sign up for Terraform Cloud and get started using the Free offering today.

Kubernetes