Mastering Terraform for_each Meta Argument

In today’s blog post, will cover terraform for_each meta argument. In particular, we will explain why we need a for_each construct, how to use for_each meta argument with examples, and what is the difference between a for_each and count meta argument.

Introduction: Why for_each?

Imagine you need to provision 10 identical EC2 instances. You could write 10 separate aws_instance blocks, but that is repetitive, error-prone, and hard to maintain. Now imagine those 10 instances are not identical – they need different names, different instance types, or different tags. Manually managing this would be a nightmare.

Historically, Terraform provided the count meta-argument for simple numerical repetition. While useful for creating fungible (interchangeable) resources, it fell short when each resource needed a unique identity or configuration.

Enter for_each. Introduced in Terraform 0.12.6, for_each revolutionized dynamic resource provisioning by allowing you to iterate over a collection of items (a map or a set of strings) and create a distinct instance of a resource, data source, or module for each item. This provides more flexibility and significantly reduces code duplication, making your configurations more scalable and maintainable.

What is the for_each Meta-Argument?

The for_each meta-argument is a special argument that can be added to any resource, data, or module block. Its primary function is to create multiple instances of that block, where each instance is uniquely identified by a key from the collection you provide.

When you use for_each, Terraform iterates over the elements of the supplied map or set of strings. And for each element, it creates a separate instance of the block, and importantly, each instance gets a distinct infrastructure object associated with it. This means you can manage each created resource independently.

Inside a block using for_each, you gain access to a special object called each. This object provides two attributes:

• each.key: Which represents the map key or the set member corresponding to the current iteration.
• each.value: Which represents the map value or (if a set was provided) the same as each.key.

for_each vs. count: The Key Differences and When to Use Each

This is one of the most sought after question in any terraform interview and sadly most of the terraform developers though have used them both in their code but can not explain the difference clearly. While both for_each and count enable creating multiple resources, their underlying mechanisms and ideal use cases differ significantly.

count Meta-Argument:

• Mechanism: Iterates a specified number of times (from “0” to “count - 1“).
• Addressing: Resources are addressed by a numerical index (e.g., aws_instance.my_server[0], aws_instance.my_server[1]).
• Stability: Less stable. If you remove an item from the middle of the list that count is based on, all subsequent resources’ indices will shift. Terraform might interpret this as needing to destroy and recreate resources, even if their underlying configuration has not changed. This can lead to unnecessary downtime or data loss.
• Use Cases:
  • Creating a fixed, small number of truly identical resources where no unique identifier is needed.
  • Conditional resource creation (e.g., count = var.enable_resource ? 1 : 0).

for_each Meta-Argument:

• Mechanism: Iterates over the elements of a map or a set of strings.
• Addressing: Resources are addressed by the map key or set member (e.g., aws_instance.my_server["web"], aws_instance.my_server["db"]).
• Stability: More stable. If you add or remove an item, only the corresponding resource is affected. The existing resources identified by other keys remain unchanged, preventing unnecessary recreation.
• Use Cases:
  • Creating resources that require unique identifiers (e.g., names, specific configurations).
  • Provisioning resources based on dynamic input collections where each item has unique attributes.
  • Managing environments (dev, staging, prod) where each environment needs a set of distinct resources.

TL;DR: Prefer for_each when your resources are not fungible and require unique identification. Use count for simple, numerical repetitions or conditional provisioning of a single resource.

How to Use for_each with Different Data Types (Examples)

for_each expects a map or a set of strings as its input. Let us look at practical examples.

Using for_each with a Set of Strings

When for_each is given a set of strings, both each.key and each.value will represent the current string in the iteration. This is useful for creating resources where the unique identifier is simply the name or a simple string.

Scenario: Create multiple S3 buckets, each with a unique name.

# variables.tf
variable "bucket_names" {
  description = "A set of unique S3 bucket names."
  type        = set(string)
  default = [
    "my-app-logs-bucket-prod",
    "my-app-data-bucket-prod",
    "my-app-backup-bucket-prod"
  ]
}

# main.tf
resource "aws_s3_bucket" "app_buckets" {
  for_each = var.bucket_names

  bucket = each.value 
  # The bucket name is the value from the set
  acl    = "private"

  tags = {
    Name        = each.key 
    # The tag name is also the key from the set
    Environment = "production"
  }
}

Explanation:

• The var.bucket_names is a set(string), ensuring unique names.
• for_each = var.bucket_names tells Terraform to create an aws_s3_bucket instance for each string in the bucket_names set.
• Inside the aws_s3_bucket block, each.value gives us the current bucket name (“my-app-logs-bucket-prod”, “my-app-data-bucket-prod”, etc.).
• each.key is identical to each.value when iterating over a set of strings.

Using for_each with a Map of Strings/Objects

Using a map is for_each‘s most powerful application, as it allows you to associate distinct configurations with each resource instance. each.key will be the map key, and each.value will be the corresponding map value.

Scenario 1: Map of Strings – Creating EC2 Instances with Different Types

# variables.tf
variable "instance_configs" {
  description = "Map of EC2 instance names to their instance types."
  type        = map(string)
  default = {
    "web-server-01" = "t2.medium",
    "db-server-01"  = "m5.large",
    "jenkins-node"  = "t3.xlarge"
  }
}

# main.tf
resource "aws_instance" "app_servers" {
  for_each = var.instance_configs

  ami           = "ami-0abcdef1234567890" # Example AMI ID
  instance_type = each.value              
  # Instance type comes from the map value (t2.medium, m5.large, t3.xlarge)
  tags = {
    Name = each.key                       
    # Server name comes from the map key (web-server-01, db-server-01, jenkins-node
  }
}

Explanation:

• var.instance_configs is a map(string) where keys are instance names and values are instance types.
• for_each = var.instance_configs creates an aws_instance for each entry.
• each.key provides the instance name (e.g., “web-server-01”), used for the Name tag.
• each.value provides the instance type (e.g., “t2.medium”).

Scenario 2: Map of Objects – More Complex Configurations

This is incredibly flexible, allowing you to pass structured data for each resource.

# variables.tf
variable "vpc_configurations" {
  description = "Configurations for different VPCs."
  type = map(object({
    cidr_block = string
    tags       = map(string)
    enable_dns_hostnames = bool
  }))
  default = {
    "development" = {
      cidr_block           = "10.10.0.0/16"
      tags = {
        Environment = "Dev"
        Project     = "MyApp"
      }
      enable_dns_hostnames = true
    },
    "production" = {
      cidr_block           = "10.20.0.0/16"
      tags = {
        Environment = "Prod"
        Project     = "MyApp"
      }
      enable_dns_hostnames = false
    }
  }
}

# main.tf
resource "aws_vpc" "my_vpcs" {
  for_each = var.vpc_configurations

  cidr_block           = each.value.cidr_block
  enable_dns_hostnames = each.value.enable_dns_hostnames

  tags = merge(
    each.value.tags,
    { "Name" = "VPC-${each.key}" } 
    # Add a dynamic Name tag
  )
}

Explanation:

• var.vpc_configurations is a map(object) where each key (e.g., “development”, “production”) maps to an object containing detailed VPC settings.
• each.value.cidr_block, each.value.tags, and each.value.enable_dns_hostnames allow direct access to the nested attributes of each object.
• The merge function is used to combine the tags defined in the variable with a dynamically generated Name tag, showing how each.key can be used for naming.

Using for_each with a List (and toset() or for expressions)

for_each strictly requires a map or a set of strings. If you have a list, you need to transform it.

Scenario 1: Converting a List of Strings to a Set with toset()

If the order does not matter and you just need unique elements, toset() is your friend.

# variables.tf
variable "availability_zones" {
  description = "List of AZs to create subnets in."
  type        = list(string)
  default     = ["ap-south-1a", "ap-south-1b"]
}

# main.tf
resource "aws_subnet" "regional_subnets" {
  for_each = toset(var.availability_zones)

  # Assuming you have an aws_vpc.main resource
  vpc_id            = aws_vpc.main.id 

  # Example CIDR, might need more complex logic
  cidr_block        = "10.0.0.0/24"   
  availability_zone = each.value

  tags = {
    Name = "subnet-${each.key}"
  }
}

Explanation:

• toset(var.availability_zones) converts the list into a set, allowing for_each to iterate over it.
• Again, each.key and each.value are the same (the AZ string).

Scenario 2: Transforming a List of Objects into a Map with a for Expression

This is common when you have a list of complex objects and need to create a unique key for for_each.

# variables.tf
variable "server_specs" {
  description = "List of server specifications."
  type = list(object({
    name          = string
    instance_type = string
    ami           = string
  }))
  default = [
    {
      name          = "web-01"
      instance_type = "t3.micro"
      ami           = "ami-0abcdef1234567890"
    },
    {
      name          = "api-01"
      instance_type = "t3.small"
      ami           = "ami-0abcdef1234567890"
    }
  ]
}

# main.tf
resource "aws_instance" "dynamic_servers" {
  # Transform list to map
  for_each = { for s in var.server_specs : s.name => s } 

  ami           = each.value.ami
  instance_type = each.value.instance_type
  tags = {
    Name = each.key
  }
}

Explanation:

• The for expression { for s in var.server_specs : s.name => s } creates a map where:
  • The key is s.name (e.g., “web-01”, “api-01”). This must be unique across your list.
  • The value is s itself (the entire object for that server).
• for_each then iterates over this newly created map.
• each.key gives you the server name, and each.value gives you the entire server object, allowing access to each.value.instance_type, each.value.ami, etc.

Advanced for_each Concepts and Use Cases

Referring to Instances

When resources are created with for_each, they form a collection, and you reference individual instances using square brackets [] with the key:

output "bucket_arns" {
  value = { for k, v in aws_s3_bucket.app_buckets : k => v.arn }
}

output "web_server_id" {
  value = aws_instance.app_servers["web-server-01"].id
}

Chaining for_each Between Resources

You can create dependencies where one for_each resource uses outputs from another, maintaining the instance-specific mapping.

Scenario: Create VPCs, and then subnets within each VPC, all dynamically.

# variables.tf
variable "network_configs" {
  type = map(object({
    vpc_cidr    = string
    # Map of subnet names to CIDRs within this VPC
    subnet_cidrs = map(string) 
  }))
  default = {
    "app-vpc" = {
      vpc_cidr = "10.0.0.0/16"
      subnet_cidrs = {
        "public"  = "10.0.1.0/24"
        "private" = "10.0.2.0/24"
      }
    },
    "db-vpc" = {
      vpc_cidr = "10.10.0.0/16"
      subnet_cidrs = {
        "data" = "10.10.1.0/24"
      }
    }
  }
}

# main.tf
resource "aws_vpc" "networks" {
  for_each = var.network_configs
  cidr_block = each.value.vpc_cidr
  tags = { Name = "VPC-${each.key}" }
}

resource "aws_subnet" "subnets" {
  # This nested for_each iterates over each VPC (each.key) and then each of its subnets (subnet_name)
  for_each = {
    for vpc_key, vpc_config in aws_vpc.networks :
    for subnet_name, subnet_cidr in vpc_config.subnet_cidrs :
    "${vpc_key}-${subnet_name}" => {
      vpc_id     = vpc_config.id
      cidr_block = subnet_cidr
      name       = subnet_name
    }
  }

  vpc_id     = each.value.vpc_id
  cidr_block = each.value.cidr_block
  tags = {
    Name = "${each.value.name}-${each.key}" 
    # Example: "public-app-vpc-public"
  }
}

Explanation:

• The aws_vpc.networks resource uses for_each to create multiple VPCs.
• The aws_subnet.subnets resource uses a complex for expression within for_each to flatten the nested map structure into a single map suitable for iteration. The key "vpc_key-subnet_name" ensures uniqueness across all subnets.
• each.value.vpc_id correctly references the ID of the parent VPC created by the aws_vpc.networks resource, maintaining the relationship.

Dynamic Blocks with for_each

dynamic blocks allow you to generate nested configuration blocks within a resource based on a collection. for_each is often used here.

Scenario: Create an EC2 instance with multiple, dynamically configured EBS volumes.

# variables.tf
variable "instance_with_volumes" {
  type = map(object({
    instance_type = string
    volumes       = list(object({
      device_name = string
      volume_size = number
    }))
  }))
  default = {
    "web-instance" = {
      instance_type = "t3.medium"
      volumes = [
        { device_name = "/dev/sdb", volume_size = 50 },
        { device_name = "/dev/sdc", volume_size = 100 }
      ]
    }
  }
}

# main.tf
resource "aws_instance" "server_with_disks" {
  for_each = var.instance_with_volumes

  ami           = "ami-0abcdef1234567890"  # Example AMI ID
  instance_type = each.value.instance_type

  tags = {
    Name = each.key
  }

  dynamic "ebs_block_device" {
    # Iterate over the 'volumes' list for each instance
    for_each = each.value.volumes 
    content {
      device_name = ebs_block_device.value.device_name
      volume_size = ebs_block_device.value.volume_size
    }
  }
}

Explanation:

• The outer for_each creates individual aws_instance resources.
• The dynamic "ebs_block_device" block then uses its own for_each to iterate over the volumes list within the current instance’s configuration.
• Inside the dynamic block, ebs_block_device.value (where ebs_block_device is the implicit iterator name) refers to the current volume object, allowing access to device_name and volume_size.

Limitations and Troubleshooting

While powerful, for_each has a few limitations:

• Values Must Be Known at Plan Time: The most common hurdle. The value provided to for_each must be known before Terraform performs any remote resource actions (i.e., during the terraform plan phase). You cannot use outputs from resources that have not been created yet as the for_each input.
  • Workaround: If you encounter “The "for_each" value depends on resource attributes that cannot be determined until apply, so Terraform cannot predict how many instances will be created.“, you might need to:
    • Split your deployment into multiple Terraform configurations (e.g., one to create the unknown value, another to use it).
    • Use terraform apply -target= to apply only the dependent resources first (not ideal for CI/CD).
    • Rethink your architecture if possible to avoid this dependency.
• Sensitive Values: for_each cannot accept sensitive values directly. This is because the values used in for_each are used to identify resource instances and will always be disclosed in UI output and state.
• List vs. Set: Remember for_each requires a map or set. Lists must be converted using toset() or for expressions to create a map.

Best Practices for Using for_each

1. Prefer for_each over count for Non-Fungible Resources: If resources have unique characteristics, identities, or lifecycles, for_each provides superior stability and manageability.
2. Use Descriptive Map Keys: When creating a map for for_each, use keys that are meaningful and uniquely identify the resource instance (e.g., environment names, server roles, unique IDs).
3. Validate Input Variables: Ensure your input variables for for_each are well-defined with appropriate types (map(string), map(object), set(string)) and include descriptions.
4. Keep for_each Expressions Simple: While for expressions can be powerful, keep the logic used to generate the for_each map or set as straightforward as possible to improve readability. Complex transformations can be handled in locals blocks first.
5. Modularize: When dynamic blocks or complex for_each scenarios become too intricate, consider extracting the logic into dedicated modules to improve organization and reusability.
6. Avoid Nested Loops in a Single Resource Block: If you find yourself needing deeply nested for_each logic within a single resource, consider if you can flatten the data structure or break it into multiple resources or modules for clarity.
7. Review terraform plan Output Carefully: Always inspect the terraform plan output when using for_each to ensure Terraform intends to create, modify, or destroy the resources as you expect, especially when making changes to the input collection.

Conclusion

The for_each meta-argument is a key component of advanced Terraform usage. By mastering its application with various data structures, you can write more precise, flexible, and robust infrastructure code. This leads to reduced boilerplate, fewer errors, and a more maintainable and scalable infrastructure-as-code repository. Embrace for_each to unlock the full potential of dynamic resource provisioning with Terraform!

Related Items:

• provider – https://ckdbtech.com/mastering-terraform-provider-meta-argument/
• depends_on – https://ckdbtech.com/mastering-terraform-depends_on-meta-argument/
• lifecycle – https://ckdbtech.com/mastering-terraform-lifecycle-meta-argument/
• count – https://ckdbtech.com/mastering-terraform-count-meta-argument/

Author

Debjeet Bhowmik

Experienced Cloud & DevOps Engineer with hands-on experience in AWS, GCP, Terraform, Ansible, ELK, Docker, Git, GitLab, Python, PowerShell, Shell, and theoretical knowledge on Azure, Kubernetes & Jenkins.
In my free time, I write blogs on ckdbtech.com