HashiCorp Terraform 0.12 Preview: For and For-Each

This is the third post of the series highlighting new features in Terraform 0.12.

Tip: As of Terraform 0.13 for_each now works with modules as well as resoruces. Follow our for_each tutorial to get started.

As part of the lead up to the release of Terraform 0.12, we are publishing a series of feature preview blog posts. The post this week is on the new iteration features: for expressions and for_each.

A common problem in Terraform configurations for versions 0.11 and earlier is dealing with situations where the number of values or resources is decided by a dynamic expression rather than a fixed count. The general problem of iteration is a big one to solve and Terraform 0.12 introduces a few different features to improve these capabilities, namely for expressions and for_each blocks. We also discuss further enhancements that will come later.

»For Expressions for List and Map Transformations

When working with lists and maps, it is common to need to apply a filter or transformation to each element in the collection and produce a new collection. Prior to Terraform 0.12, Terraform only had limited support for such operations via a few tailored interpolation functions, such as formatlist.

Terraform 0.12 introduces a new construct called a for expression, which allows the construction of a list or map by transforming and filtering elements in another list or map. The example below shows this in use:

# Configuration for Terraform 0.12

variable "vpc_id" {
  description = "ID for the AWS VPC where a security group is to be created."
}

variable "subnet_numbers" {
  description = "List of 8-bit numbers of subnets of base_cidr_block that should be granted access."
  default = [1, 2, 3]
}

data "aws_vpc" "example" {
  id = var.vpc_id
}

resource "aws_security_group" "example" {
  name        = "friendly_subnets"
  description = "Allows access from friendly subnets"
  vpc_id      = var.vpc_id

  ingress {
    from_port = 0
    to_port   = 0
    protocol  = -1

    # For each number in subnet_numbers, extend the CIDR prefix of the
    # requested VPC to produce a subnet CIDR prefix.
    # For the default value of subnet_numbers above and a VPC CIDR prefix
    # of 10.1.0.0/16, this would produce:
    #   ["10.1.1.0/24", "10.1.2.0/24", "10.1.3.0/24"]
    cidr_blocks = [
      for num in var.subnet_numbers:
      cidrsubnet(data.aws_vpc.example.cidr_block, 8, num)
    ]
  }
}

When a for expression is wrapped in square brackets ([ and ]) as shown above, the result is a list. A for expression wrapped in braces ({ and }) produces a map in a similar way. For example:

# Configuration for Terraform 0.12

output "instance_private_ip_addresses" {
  # Result is a map from instance id to private IP address, such as:
  #  {"i-1234" = "192.168.1.2", "i-5678" = "192.168.1.5"}
  value = {
    for instance in aws_instance.example:
    instance.id => instance.private_ip
  }
}

The optional if clause can be used also to filter the input collection so that the result contains only a subset of input items:

# Configuration for Terraform 0.12

output "instance_public_ip_addresses" {
  value = {
    for instance in aws_instance.example:
    instance.id => instance.public
    if instance.associate_public_ip_address
  }
}

Finally, the map form of a for expression has a grouping mode where the map key is used to group items together into a list for each distinct key. This mode is activated by placing an ellipsis (...) after the value expression:

# Configuration for Terraform 0.12

output "instances_by_availability_zone" {
  # Result is a map from availability zone to instance ids, such as:
  #  {"us-east-1a": ["i-1234", "i-5678"]}
  value = {
    for instance in aws_instance.example:
    instance.availability_zone => instance.id...
  }
}

In the the example above we show that a resource with count set now also behaves as a list, allowing all of the instances of aws_instance.example to be iterated over to produce a grouping by availability zone.

These new expressions can be used to generate a value for any argument that expects a list or map expression. And in Terraform 0.12, this includes any location that accepts a list or map, including module inputs.

»Dynamic Nested Blocks

Several resource types use nested configuration blocks to define repeatable portions of their configuration. Terraform 0.12 introduces a new construct for dynamically constructing a collection of nested configuration blocks.

For example, the aws_autoscaling_group resource type uses nested blocks to declare tags that may or may not be propagated to any created EC2 instances. The example below shows the Terraform 0.11 and earlier syntax:

# Configuration for Terraform 0.11 and earlier

resource "aws_autoscaling_group" "example" {
  # ...

  tag {
    key                 = "Name"
    value               = "example-asg-name"
    propagate_at_launch = false
  }

  tag {
    key                 = "Component"
    value               = "user-service"
    propagate_at_launch = true
  }

  tag {
    key                 = "Environment"
    value               = "production"
    propagate_at_launch = true
  }
}

Because these nested blocks are validated statically, it was previously difficult or impossible to implement more dynamic behaviors. Some users found ways to exploit some implementation details to trick Terraform into partially supporting dynamic generation of blocks, but these workarounds were unreliable because Terraform makes assumptions about nested blocks that do not hold for arbitrary expressions.

Terraform 0.12 introduces a special new dynamic block construct to support these dynamic configuration use-cases in a first-class way. The same example converted to Terraform 0.12:

# Configuration for Terraform 0.12

locals {
  standard_tags = {
    Component   = "user-service"
    Environment = "production"
  }
}

resource "aws_autoscaling_group" "example" {
  # ...

  tag {
    key                 = "Name"
    value               = "example-asg-name"
    propagate_at_launch = false
  }

  dynamic "tag" {
    for_each = local.standard_tags
		
    content {
      key                 = tag.key
      value               = tag.value
      propagate_at_launch = true
    }
  }
}

A dynamic "tag" block behaves as if a separate tag block were written for each element of the list or map given in the for_each argument. Because dynamic is itself given as a nested block, all of the same syntax constructs can be used within its content block that would normally be valid in a literal tag block, and both static and dynamic tag blocks can be mixed as shown above. This enables arbitrarily complex behaviors as necessary.

Since the for_each argument accepts any list or map expression, this feature can be combined with for expressions as described above to create nested blocks based on arbitrary transformations of other list and map values:

# Configuration for Terraform v0.12

variable "subnets" {
  default = [
    {
      name   = "a"
      number = 1
    },
    {
      name   = "b"
      number = 2
    },
    {
      name   = "c"
      number = 3
    },
  ]
}

locals {
  base_cidr_block = "10.0.0.0/16"
}

resource "azurerm_virtual_network" "example" {
  name                = "example-network"
  resource_group_name = azurerm_resource_group.test.name
  address_space       = [local.base_cidr_block]
  location            = "West US"

  dynamic "subnet" {
    for_each = [for s in subnets: {
      name   = s.name
      prefix = cidrsubnet(local.base_cidr_block, 4, s.number)
    }]
	 
    content {
      name           = subnet.value.name
      address_prefix = subnet.value.prefix
    }
  }
}

Terraform is able to validate the arguments inside the content block in the same way as it would validate a static block, even if the value of the for_each expression is not yet known. Thus this approach allows more problems to be caught at plan time, before any real actions have been taken.

We still recommend that you avoid writing overly-abstract, dynamic configurations as a general rule. These dynamic features can be useful when creating reusable modules, but over-use of dynamic behavior will hurt readability and maintainability. Explicit is better than implicit, and direct is better than indirect.

»Resource for_each

The dynamic block construct described previously includes the idea of iterating over a list or map using the for_each argument, which is intended as a more intuitive and useful way to create dynamic nested blocks compared to the count argument on resources.

During the development of Terraform 0.12 we've also laid the groundwork for supporting for_each directly inside a resource or data block as a more convenient way to create a resource instance for each element in a list or map. Unfortunately we will not be able to fully complete this feature for the Terraform 0.12 initial release, but we plan to include this in a subsequent release to make it easier to dynamically construct multiple resource instances of the same type based on elements of a given map. The example below shows future Terraform syntax, not included in the initial 0.12 release:

# Planned for a later release, after v0.12.0

variable "subnet_numbers" {
  description = "Map from availability zone to the number that should be used for each availability zone's subnet"
  default     = {
    "eu-west-1a" = 1
    "eu-west-1b" = 2
    "eu-west-1c" = 3
  }
}

resource "aws_vpc" "example" {
  # ...
}

resource "aws_subnet" "example" {
  for_each = var.subnet_numbers

  vpc_id            = aws_vpc.example.id
  availability_zone = each.key
  cidr_block        = cidrsubnet(aws_vpc.example.cidr_block, 8, each.value)
}

The new object each, with attributes each.key and each.value, will allow access to the key and value of each element of the for_each expression, in a similar way as count.index for the count argument.

When the for_each argument value is a map, Terraform will identify each instance by the string key of the map element rather than by a numeric index, which will avoid many limitations with the current pattern of using count to iterate over a list where items may be added and removed from the middle of that list, changing the subsequent indices.

While we will not be able to complete this feature in time for the initial Terraform 0.12 release, Terraform will consider the argument name for_each and the expression each to be reserved and not usable by resource types. That means that this feature can be completed in a subsequent release without additional breaking changes.

»Module count and for_each

For a long time, users have wished to be able to use the count meta-argument within module blocks, allowing multiple instances of the same module to be created more easily.

Again, we have been laying the groundwork for this during Terraform 0.12 development and expect to complete this work in a later release. Along with count, module blocks will also accept the new for_each argument described for resources above, with similar results.

This feature is particularly complicated to implement within Terraform's existing architecture, so some more work will certainly be required before we can support this. To avoid further breaking changes in later releases, 0.12 will reserve the module input variable names count and for_each in preparation for the completion of this feature.

»Upgrade Guide

The new for and for_each functionality introduces new reserved words into existing resources and modules. We've verified that this doesn't introduce any breaking changes to the resources provided by official providers. User-created modules that use count or for_each will need to be updated.

The future functionality (beyond Terraform 0.12) described in this post for resource and module iteration will not introduce any further breaking changes since we are reserving all the necessary keywords in the Terraform 0.12 release.

The existing count functionality remains working as before, but many edge cases have been resolved by the underlying architecture changes to support arbitrary for_each.

»Next Steps

This was part 3 of the blog post series previewing Terraform 0.12.

for expressions and for_each will be released in Terraform 0.12 (except those use cases specifically noted), coming later this summer. To learn more about how to upgrade to Terraform 0.12, read the upgrade instructions which will be continuously updated as we get closer to releasing Terraform 0.12. If you have any feedback or concerns about these changes, please communicate with the Terraform team via the public mailing list.