Announcing Cost Estimation for Terraform Cloud and Enterprise

Today we're releasing Cost Estimation for HashiCorp Terraform Cloud and HashiCorp Terraform Enterprise. Cost Estimation provides organizations insight into the cost implication of infrastructure changes before they’re applied. Included in this release is integration with policy as code workflows via Sentinel to enable preventative cost-control policies to be automatically assessed against changes.

Organizations that are using an infrastructure as code approach to manage their multi-cloud provisioning, compliance, and management requirements can find it challenging to understand the cost implications of a change before it is applied. Many have been relying on after-the-fact alerts from their cloud provider, using dedicated third party services that continually monitor changes in cost, or potentially waiting until they receive their end of month bill to understand the cost impact of their changes. This new capability now enables teams who manage their self-service infrastructure to view an estimate of changes in monthly cost from their cloud provider before applying any change. HashiCorp’s Sentinel allows cost-centric policies to be created and then automatically enforced in the Terraform workflow. Administrators then have the ability to approve significant changes or to completely prevent specific workspaces from exceeding predetermined thresholds.

This blog will discuss the cost estimation capability and the workflow including:

• How to enable Cost Estimation
• Understanding the cost impact of changes
• Escalation workflows for expensive changes
• Applying strict cost controls and automated oversight

»Pre-emptive cost visibility

One common challenge of cloud infrastructure adoption is enabling the practitioners deploying the changes to understand the financial impact of the changes they’re applying. Many no longer require direct access to the console of the cloud provider, so they don’t see the billing related details of the resources they are provisioning until long after those resources have been deployed. This can create a situation where those responsible for financial governance need to work with DevOps teams to retrospectively reduce the cost profile of infrastructure after it has been deployed. A task that is now more complicated and carries more risk than if there had been earlier intervention.

Addressing it from a more proactive standpoint, organizations can take the shift left approach: IT Ops researches problems that impact cost, collate and collect the data to formalize policies and influence actions across all DevOps teams, and enable everyone in the organization to take consistent actions based estimated infrastructure costs and company policy.

All teams using Terraform Cloud and Enterprise can now see an estimate of changes before they are applied by enabling “Cost Estimation” within the settings of their Terraform Organization:

»Understanding the cost impact of changes

Once Cost Estimation is applied to an Terraform organization, all future runs will include an estimate of the updated total monthly cost for the resources that have changed, along with a separate estimate of the change in cost:

In the above example, you can see that the total monthly cost has increased by $14.98/mo, which brings the total cost of those resources up to $37.44/mo. This change increases the maximum allowed size of an Auto Scaling Group on AWS from 3 instances to 5. A simplified example of that config looks like:

resource "aws_autoscaling_group" "prod-web-servers" {
  name                      = "prod-web-servers"
  max_size                  = 5
  min_size                  = 0
  desired_capacity          = 0
}

It’s worth pointing out that it is only the max_size argument which has changed here, the min_size and desired_capacity have remained as zero. While in effect this most likely means there is no immediate cost impact from this change (in this example zero is the desired capacity so there should be no instances running). The other consideration is that dynamic scaling rules could change this at any time up to the maximum threshold. As a result the estimate takes the maximum potential footprint into consideration when calculating the change.

Now practitioners applying a change can have before-the-fact visibility into the potential cost impact of a change. This makes it easier to identify simple mistakes in configuration that could have significant financial implications, collaborate with other business units to keep costs aligned with forecasts, and support early intervention and remediation workflows at the most cost effective and lowest risk time to adjust implementation.

»Enforcing cost controls with HashiCorp Sentinel

Sentinel is a Policy as Code framework that’s integrated into multiple product offerings from HashiCorp. Sentinel within Terraform enables an organization to define policies that are enforced against infrastructure between the plan and apply phases of a Terraform run. Compared to many tools that scan existing infrastructure for policy infractions, Sentinel proactively prevents provisioning of out-of-policy infrastructure and gives teams the confidence that all changes they deploy and within the organization’s policy

These latest enhancements with cost estimation expand on that capability to ensure consistent financial governance is applied to infrastructure changes.

»Escalation workflows for expensive changes

It can be difficult to find the right pragmatic balance between allowing teams the agility to provision the infrastructure they need with keeping costs aligned with the expected value of a project. This can then lead to approval workflows that require oversight from an individual or team to determine if changes in cost are reasonable, which in turn can slow delivery and in itself increase the cost of implementation.

A Policy as Code approach that takes advantage of the cost estimation features of Terraform means organizations can now set guidelines on what is an acceptable change that requires review and then only escalate when a change is in breach of the standard policy. This frees up time in the approval workflow by ensuring that team is only required to review genuine escalations, and the practitioners responsible for implementation are able to self-service in-policy changes with confidence.

In the example below you see a Terraform run which has breached a “soft” policy check:

You can see the failed state here is against the cost-control/modest-increases policy, and the entire run has now been halted and placed into a “Policy Override” state. This is because this policy has been written to prevent any single change that increases the cost more than $500/mo.

For this run to proceed, a Terraform user with admin or policy author permissions will be required to review the plan, provide an optional explanation for why the policy is being overridden, and then ultimately click “Override & Continue”.

The code to implement the policy is:

import "tfrun"
import "decimal"

main = rule {
  decimal.new(tfrun.cost_estimate.delta_monthly_cost).less_than(500)
}

In this example, the recently announced tfrun import is used, along with the new decimal import to ensure working with data types that are consistent for dealing with currency. The next part is to access the change in monthly cost that has been estimated for this run ( tfrun.cost_estimate.delta_monthly_cost ) and ensure it is less than 500.

»Applying strict cost controls and automated oversight

There may also been known cost thresholds that should never be breached, and an automatic escalation into an approval workflow where a review would be unnecessary. An example could be preventing workspaces that are managing “development” environments from ever exceeding a maximum estimated monthly cost. With such a policy in place, developers have the freedom and confidence to experiment with any infrastructure configuration they desire without the risk of uncomfortable conversations later. It also expands their autonomy in self-service workflows as a breach in policy allows them to make pragmatic decisions about what infrastructure may be able to be deprovisioned to free additional budget for the new changes they wish to deploy.

Here you can see an example of a Terraform run that has had a policy “hard” fail:

The hard failure in this instance is the cost-control/max-budget policy which has been defined to prevent the total estimated monthly cost from exceeding $10,000/mo. The code to implement this policy is:

import "tfrun"
import "decimal"

main = rule {
  decimal.new(tfrun.cost_estimate.proposed_monthly_cost).less_than(10000)
}

The code here is very similar to the previous example with two notable differences:

• the value of tfrun.cost_estimate.proposed_monthly_cost which provides an estimated aggregate cost of the resources running in the workspace, rather than just the expected change in total cost.
• the comparison value is updated to 10,000 (up from 500).

The video below demonstrates the cost estimation workflow with Sentinel policies and enforcement:

»Getting Started

You’ve seen just a glimpse of the workflows supported by the cost estimation and policy features available as part of Terraform Cloud and Enterprise, and how they enable teams to self-service changes in dynamic environments while also giving them the confidence that they will stay within guidelines set by their organization.

For more information on Terraform Cloud and Terraform Enterprise or to get started with your free trial, visit the Terraform product page. To learn more about Terraform visit the HashiCorp Learn platform and see it in action.