Terraform at Scale: How We Manage 500+ Resources Across 8 AWS Accounts Without Losing Our Minds

November 2026 | ~18 min read

The `terraform destroy` That Took Down Staging

It was 3:47 PM on a Thursday. An engineer ran terraform destroy to clean up a sandbox VPC. Except they were in the wrong Terraform workspace. Staging was gone — 47 resources deleted in 90 seconds. The RDS instance. The EKS cluster. The VPN connections. All of it.

Recovery took 6 hours. Not because the infrastructure was hard to rebuild — we had the Terraform code. But because nobody could figure out what the staging state file actually contained. The state was a single 50MB monolith covering every resource across every environment. Running terraform plan took 15 minutes. Finding which resources belonged to staging vs. production vs. shared-services required reading through 12,000 lines of state JSON.

The real problem wasn't the accidental destroy. It was everything that made it possible:

One monolithic state file covering 500+ resources across 8 AWS accounts
No CI/CD for Terraform — every engineer applied from their laptop
Constant state lock conflicts — 6 engineers fighting for the same DynamoDB lock 12 times a week
No blast radius controls — a single terraform apply could touch any resource in any account
No drift detection — console changes piled up silently until the next plan showed 40 unexpected diffs
15-minute plan times — so engineers stopped running plans before applying

We rebuilt the entire Terraform workflow in 4 weeks. Here's everything.

The Numbers

┌──────────────────────────────────────────────────────────────────────┐ │ TERRAFORM AT SCALE RESULTS │ ├───────────────────────────────────────┬────────────┬─────────────────┤ │ Metric │ Before │ After │ ├───────────────────────────────────────┼────────────┼─────────────────┤ │ Resources managed │ 500+ │ 500+ │ │ AWS accounts │ 8 │ 8 │ │ State files │ 1 (50MB) │ 24 (<3MB each) │ │ Plan time │ 15 min │ 45 sec │ │ State lock conflicts/week │ 12 │ 0 │ │ Drift incidents/quarter │ 8 │ 0 │ │ Time to apply a change │ 45 min │ 8 min │ │ Blast radius (max resources/apply) │ 500+ │ ~30 │ │ Accidental destroys (6 month window) │ 2 │ 0 │ │ Engineer onboarding time │ 2 weeks │ 2 hours │ │ Engineers applying from laptops │ 6 │ 0 │ └───────────────────────────────────────┴────────────┴─────────────────┘

Zero additional AWS cost. This was purely an operational improvement — better tooling, better structure, better workflows.

The Monolith Problem
State Architecture
Multi-Account Strategy
Module Composition Patterns
CI/CD for Terraform
Blast Radius Controls
Drift Detection and Reconciliation
Developer Experience
Lessons Learned
Action Items

The Monolith Problem

Here's what our Terraform looked like before:

infrastructure/ ├── main.tf # 2,400 lines ├── variables.tf # 800 lines ├── outputs.tf # 400 lines ├── terraform.tfstate # 50MB, 12,000 lines └── terraform.tfvars # 200 lines, workspace-switched

One directory. One state file. Every resource — VPCs, subnets, security groups, RDS instances, EKS clusters, S3 buckets, IAM roles, CloudFront distributions, Lambda functions — all in the same Terraform root. Workspaces for environments.

The symptoms were predictable:

15-minute plans. Every terraform plan refreshed 500+ resources across 8 AWS accounts. Engineers started skipping plans and running terraform apply directly. "It was fine last time" is not a deployment strategy.

Constant state locks. Six engineers, one DynamoDB lock. Someone runs a plan, goes to lunch, forgets to cancel. Everyone else is blocked. We had a Slack channel called #terraform-lock where people asked "who has the lock?" twelve times a week.

50MB state file. Every terraform plan downloaded 50MB from S3, parsed 12,000 lines of JSON, made 500+ API calls to check resource status, and uploaded the result back. The state file contained secrets (RDS passwords, API keys) that every engineer could read.

Maximum blast radius. A single terraform apply could modify any resource in any account. There was no isolation between the VPC that routes production traffic and the S3 bucket that stores dev logs. One bad variable change could cascade.

The root cause was simple: one state file = one blast radius = maximum risk.

┌──────────────────────────────────────────────────────────────────┐ │ THE MONOLITH STATE │ │ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ VPC │ │ EKS │ │ RDS │ │ IAM │ │ │ │ (prod) │ │ (prod) │ │ (prod) │ │ (global)│ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ VPC │ │ EKS │ │ RDS │ │ S3 │ │ │ │(staging)│ │(staging)│ │(staging)│ │ (all) │ │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ ┌─────────┐ ┌─────────┐ ┌─────────┐ ┌─────────┐ │ │ │ Lambda │ │CloudFrnt│ │ Route53│ │ ... │ │ │ │ (all) │ │ (all) │ │ (all) │ │ (all) │ 500+ total │ │ └─────────┘ └─────────┘ └─────────┘ └─────────┘ │ │ │ │ ONE state lock. ONE blast radius. ONE bad day. │ └──────────────────────────────────────────────────────────────────┘

The fix wasn't workspaces (we'd tried that — same state structure, different instances). It was splitting the state by domain and environment.

State Architecture

We split the monolith into 24 state files organized by domain and environment:

infrastructure/ ├── networking/ │ ├── prod/ │ │ ├── main.tf │ │ ├── variables.tf │ │ └── backend.tf # → s3://tf-state/networking/prod/terraform.tfstate │ └── staging/ │ ├── main.tf │ ├── variables.tf │ └── backend.tf # → s3://tf-state/networking/staging/terraform.tfstate ├── eks/ │ ├── prod/ │ └── staging/ ├── data/ │ ├── prod/ # RDS, ElastiCache, OpenSearch │ └── staging/ ├── apps/ │ ├── payments-prod/ │ ├── payments-staging/ │ ├── platform-prod/ │ └── platform-staging/ ├── security/ │ └── global/ # IAM roles, KMS keys, GuardDuty ├── dns/ │ └── global/ # Route53 zones, records └── modules/ # Shared modules (versioned via git tags) ├── vpc/ ├── eks-cluster/ ├── rds-instance/ └── service-deployment/

Each leaf directory is an independent Terraform root with its own state file, its own lock, and its own blast radius.

The backend configuration uses a consistent pattern:

terraform { backend "s3" { bucket = "company-terraform-state" key = "networking/prod/terraform.tfstate" region = "us-east-1" dynamodb_table = "terraform-locks" encrypt = true } }

The state bucket itself is managed by a bootstrap root that's applied once and rarely touched:

resource "aws_s3_bucket" "terraform_state" { bucket = "company-terraform-state" } resource "aws_s3_bucket_versioning" "terraform_state" { bucket = aws_s3_bucket.terraform_state.id versioning_configuration { status = "Enabled" } } resource "aws_s3_bucket_server_side_encryption_configuration" "terraform_state" { bucket = aws_s3_bucket.terraform_state.id rule { apply_server_side_encryption_by_default { sse_algorithm = "aws:kms" } } } resource "aws_s3_bucket_public_access_block" "terraform_state" { bucket = aws_s3_bucket.terraform_state.id block_public_acls = true block_public_policy = true ignore_public_acls = true restrict_public_buckets = true } resource "aws_dynamodb_table" "terraform_locks" { name = "terraform-locks" billing_mode = "PAY_PER_REQUEST" hash_key = "LockID" attribute { name = "LockID" type = "S" } }

Cross-state references use terraform_remote_state data sources or SSM Parameter Store. We prefer SSM because it decouples state files — a networking change doesn't require re-planning the EKS root:

# In networking/prod — publish VPC ID resource "aws_ssm_parameter" "vpc_id" { name = "/infrastructure/networking/prod/vpc_id" type = "String" value = module.vpc.vpc_id } # In eks/prod — consume VPC ID data "aws_ssm_parameter" "vpc_id" { name = "/infrastructure/networking/prod/vpc_id" } module "eks" { source = "../../modules/eks-cluster" vpc_id = data.aws_ssm_parameter.vpc_id.value # ... }

State migration from the monolith was the hardest part. We used terraform state mv to move resources one domain at a time, starting with the lowest-risk domain (DNS):

# Move Route53 resources to the new dns/global state terraform state mv \ -state=terraform.tfstate \ -state-out=dns/global/terraform.tfstate \ 'aws_route53_zone.primary' \ 'aws_route53_zone.primary' terraform state mv \ -state=terraform.tfstate \ -state-out=dns/global/terraform.tfstate \ 'aws_route53_record.apex' \ 'aws_route53_record.apex'

We migrated one domain per day over two weeks, running terraform plan in both the source and destination after each move to confirm zero diff.

Multi-Account Strategy

Eight AWS accounts, each with a dedicated purpose:

┌──────────────────────────────────────────────────────────────┐ │ AWS ORGANIZATIONS │ │ │ │ ┌─────────────┐ │ │ │ Management │ ← Organization root, billing, SCPs │ │ └──────┬──────┘ │ │ │ │ │ ┌────┴─────────────────────────────────┐ │ │ │ │ │ │ ┌─┴──────────┐ ┌────────────┐ ┌───────┴──────┐ │ │ │ Security │ │ Shared │ │ Workloads │ │ │ │ Account │ │ Services │ │ OU │ │ │ │ │ │ │ │ │ │ │ │ GuardDuty │ │ ECR │ │ ┌──────────┐│ │ │ │ Config │ │ Terraform │ │ │ Prod ││ │ │ │ CloudTrail │ │ State │ │ ├──────────┤│ │ │ │ IAM IdC │ │ CI/CD │ │ │ Staging ││ │ │ └────────────┘ │ Artifacts │ │ ├──────────┤│ │ │ └────────────┘ │ │ Dev ││ │ │ │ ├──────────┤│ │ │ │ │ Sandbox ││ │ │ │ ├──────────┤│ │ │ │ │ Data ││ │ │ │ └──────────┘│ │ │ └─────────────┘ │ └──────────────────────────────────────────────────────────────┘

Terraform runs in the shared-services account and assumes roles into target accounts:

locals { account_ids = { management = "111111111111" security = "222222222222" shared_services = "333333333333" prod = "444444444444" staging = "555555555555" dev = "666666666666" sandbox = "777777777777" data = "888888888888" } } provider "aws" { region = "us-east-1" assume_role { role_arn = "arn:aws:iam::${local.account_ids[var.environment]}:role/TerraformExecutionRole" } default_tags { tags = { ManagedBy = "terraform" Environment = var.environment Domain = var.domain } } }

The TerraformExecutionRole in each account trusts the shared-services account and has scoped permissions — not AdministratorAccess:

resource "aws_iam_role" "terraform_execution" { name = "TerraformExecutionRole" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { AWS = "arn:aws:iam::${local.account_ids.shared_services}:root" } Action = "sts:AssumeRole" Condition = { StringEquals = { "aws:PrincipalTag/team" = "platform" } } }] }) } resource "aws_iam_role_policy_attachment" "terraform_execution" { role = aws_iam_role.terraform_execution.name policy_arn = "arn:aws:iam::aws:policy/PowerUserAccess" }

We use PowerUserAccess instead of AdministratorAccess because Terraform shouldn't manage IAM policies in workload accounts — that's the security account's job.

Module Composition Patterns

We use a three-tier module architecture:

┌─────────────────────────────────────────────────────────────┐ │ Tier 3: Composition Roots (environment-specific) │ │ infrastructure/eks/prod/main.tf │ │ infrastructure/apps/payments-prod/main.tf │ │ These call tier-2 modules with environment-specific values │ ├─────────────────────────────────────────────────────────────┤ │ Tier 2: Service Modules (business logic) │ │ modules/eks-cluster/ modules/service-deployment/ │ │ modules/rds-instance/ modules/static-site/ │ │ These compose tier-1 resources into business units │ ├─────────────────────────────────────────────────────────────┤ │ Tier 1: Infrastructure Resources (AWS primitives) │ │ aws_eks_cluster aws_rds_cluster │ │ aws_vpc aws_s3_bucket │ │ Raw Terraform resources — no abstraction │ └─────────────────────────────────────────────────────────────┘

Module interface design: Inputs should be business concepts, not AWS implementation details. Bad: instance_class, engine_version, allocated_storage. Good: size (small/medium/large), high_availability (bool), environment (string).

# modules/rds-instance/variables.tf variable "name" { type = string description = "Database name (used for identifier prefix and tags)" } variable "size" { type = string description = "T-shirt size: small, medium, large, xlarge" validation { condition = contains(["small", "medium", "large", "xlarge"], var.size) error_message = "Size must be small, medium, large, or xlarge." } } variable "high_availability" { type = bool description = "Enable Multi-AZ deployment" default = false } variable "backup_retention_days" { type = number description = "Number of days to retain automated backups" default = 7 }

# modules/rds-instance/main.tf locals { size_map = { small = { instance_class = "db.t4g.medium", storage = 50, iops = 3000 } medium = { instance_class = "db.r6g.large", storage = 100, iops = 6000 } large = { instance_class = "db.r6g.xlarge", storage = 500, iops = 12000 } xlarge = { instance_class = "db.r6g.2xlarge", storage = 1000, iops = 20000 } } config = local.size_map[var.size] } resource "aws_db_instance" "this" { identifier_prefix = "${var.name}-" instance_class = local.config.instance_class allocated_storage = local.config.storage iops = local.config.iops storage_type = "gp3" engine = "postgres" engine_version = "16.4" multi_az = var.high_availability backup_retention_period = var.backup_retention_days deletion_protection = var.high_availability skip_final_snapshot = !var.high_availability final_snapshot_identifier = var.high_availability ? "${var.name}-final" : null copy_tags_to_snapshot = true tags = { Name = var.name } }

Versioning: Modules are versioned via git tags. Composition roots pin to specific versions:

# infrastructure/data/prod/main.tf module "payments_db" { source = "git::https://github.com/company/terraform-modules.git//rds-instance?ref=v2.3.1" name = "payments" size = "large" high_availability = true }

Never use ref=main in production. A module change in main shouldn't automatically propagate to every environment. Upgrade versions deliberately: staging first, wait a week, then production.

Anti-patterns we eliminated:

Modules calling modules calling modules (max depth: 2)
Passing 40+ variables through a module that just forwards them
count and for_each on modules (prefer separate module calls for clarity)
Local path module sources in production (impossible to version or roll back)

CI/CD for Terraform

The single biggest improvement. We went from "apply from laptops" to "nobody touches terraform apply directly."

┌────────────────────────────────────────────────────────────┐ │ TERRAFORM CI/CD FLOW │ │ │ │ Engineer │ │ │ │ │ ├── Push branch │ │ │ │ │ │ │ ▼ │ │ │ GitHub Actions: terraform plan │ │ │ │ │ │ │ ▼ │ │ │ Plan output posted as PR comment │ │ │ │ │ │ ├── Open PR, request review │ │ │ │ │ │ │ ▼ │ │ │ Reviewer approves (reviews plan output) │ │ │ │ │ │ ├── Merge to main │ │ │ │ │ │ │ ▼ │ │ │ GitHub Actions: terraform apply │ │ │ │ │ │ │ ▼ │ │ │ Slack notification with apply result │ │ │ │ │ Nobody runs terraform apply locally. Ever. │ └────────────────────────────────────────────────────────────┘

The GitHub Actions workflow detects which state files changed and only plans/applies those:

# .github/workflows/terraform.yml name: Terraform on: pull_request: paths: - 'infrastructure/**' push: branches: [main] paths: - 'infrastructure/**' permissions: id-token: write contents: read pull-requests: write jobs: detect-changes: runs-on: ubuntu-latest outputs: directories: ${{ steps.changed.outputs.directories }} steps: - uses: actions/checkout@v4 with: fetch-depth: 0 - id: changed run: | dirs=$(git diff --name-only ${{ github.event.before }} ${{ github.sha }} \ | grep '^infrastructure/' \ | grep -v '^infrastructure/modules/' \ | xargs -I{} dirname {} \ | sort -u \ | jq -R -s -c 'split("\n") | map(select(length > 0))') echo "directories=$dirs" >> "$GITHUB_OUTPUT" plan: needs: detect-changes if: github.event_name == 'pull_request' runs-on: ubuntu-latest strategy: matrix: directory: ${{ fromJson(needs.detect-changes.outputs.directories) }} steps: - uses: actions/checkout@v4 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::333333333333:role/GitHubActionsTerraform aws-region: us-east-1 - uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.9.0 - name: Terraform Init working-directory: ${{ matrix.directory }} run: terraform init -input=false - name: Terraform Plan id: plan working-directory: ${{ matrix.directory }} run: terraform plan -input=false -no-color -out=tfplan continue-on-error: true - name: Comment Plan on PR uses: actions/github-script@v7 with: script: | const output = `#### Terraform Plan: \`${{ matrix.directory }}\` \`\`\` ${{ steps.plan.outputs.stdout }} \`\`\` *Plan: ${{ steps.plan.outcome }}*`; github.rest.issues.createComment({ issue_number: context.issue.number, owner: context.repo.owner, repo: context.repo.repo, body: output.substring(0, 65536) }); - name: Fail on Plan Error if: steps.plan.outcome == 'failure' run: exit 1 apply: needs: detect-changes if: github.ref == 'refs/heads/main' && github.event_name == 'push' runs-on: ubuntu-latest strategy: max-parallel: 1 matrix: directory: ${{ fromJson(needs.detect-changes.outputs.directories) }} steps: - uses: actions/checkout@v4 - uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::333333333333:role/GitHubActionsTerraform aws-region: us-east-1 - uses: hashicorp/setup-terraform@v3 with: terraform_version: 1.9.0 - name: Terraform Init working-directory: ${{ matrix.directory }} run: terraform init -input=false - name: Terraform Apply working-directory: ${{ matrix.directory }} run: terraform apply -input=false -auto-approve

OIDC federation — no static AWS keys in GitHub Secrets:

resource "aws_iam_openid_connect_provider" "github_actions" { url = "https://token.actions.githubusercontent.com" client_id_list = ["sts.amazonaws.com"] thumbprint_list = ["6938fd4d98bab03faadb97b34396831e3780aea1"] } resource "aws_iam_role" "github_actions_terraform" { name = "GitHubActionsTerraform" assume_role_policy = jsonencode({ Version = "2012-10-17" Statement = [{ Effect = "Allow" Principal = { Federated = aws_iam_openid_connect_provider.github_actions.arn } Action = "sts:AssumeRoleWithWebIdentity" Condition = { StringEquals = { "token.actions.githubusercontent.com:aud" = "sts.amazonaws.com" } StringLike = { "token.actions.githubusercontent.com:sub" = "repo:company/infrastructure:*" } } }] }) }

Blast Radius Controls

State splitting handles the biggest risk, but we added three more layers:

1. Lifecycle prevent_destroy on critical resources:

resource "aws_db_instance" "production" { # ... configuration ... lifecycle { prevent_destroy = true } } resource "aws_s3_bucket" "data_lake" { bucket = "company-data-lake-prod" lifecycle { prevent_destroy = true } }

If someone accidentally adds a change that would destroy these resources, terraform plan fails instead of showing a destroy action.

2. OPA policies via Conftest:

We run Open Policy Agent checks on the plan output before apply:

# policy/terraform.rego package terraform deny[msg] { resource := input.resource_changes[_] resource.change.actions[_] == "delete" resource.address == "aws_db_instance.production" msg := sprintf("BLOCKED: Cannot destroy production database %s", [resource.address]) } deny[msg] { resource := input.resource_changes[_] resource.type == "aws_security_group_rule" resource.change.after.cidr_blocks[_] == "0.0.0.0/0" resource.change.after.type == "ingress" msg := sprintf("BLOCKED: Security group %s allows ingress from 0.0.0.0/0", [resource.address]) } deny[msg] { resource := input.resource_changes[_] resource.type == "aws_s3_bucket" not resource.change.after.server_side_encryption_configuration msg := sprintf("BLOCKED: S3 bucket %s missing encryption configuration", [resource.address]) }

The CI pipeline runs conftest test on the plan JSON output:

 - name: Policy Check working-directory: ${{ matrix.directory }} run: | terraform show -json tfplan > plan.json conftest test plan.json --policy ../../policy/

3. Infracost for plan-time cost estimation:

Every PR gets a cost estimate comment so reviewers can catch runaway spend before merge:

 - name: Infracost uses: infracost/actions/setup@v3 with: api-key: ${{ secrets.INFRACOST_API_KEY }} - name: Generate Cost Estimate run: | infracost breakdown \ --path=${{ matrix.directory }} \ --format=json \ --out-file=/tmp/infracost.json - name: Post Cost Comment run: | infracost comment github \ --path=/tmp/infracost.json \ --repo=${{ github.repository }} \ --pull-request=${{ github.event.pull_request.number }} \ --github-token=${{ secrets.GITHUB_TOKEN }}

Drift Detection and Reconciliation

Console changes are the silent killer. Someone tweaks a security group in the AWS console during an incident. Someone enables a feature flag in RDS settings. Someone changes a tag for a one-off report. None of it goes through Terraform. The state slowly diverges from reality.

We run nightly drift detection across all 24 state files:

# .github/workflows/drift-detection.yml nameSource：https://dev.to/garrett_yan/terraform-at-scale-how-we-manage-500-resources-across-8-aws-accounts-without-losing-our-minds-39j6My favorites

PreviousGitHub Actions Is Changing pull_request_target — Check Your Workflows Before December 2NextSolanaGuard AI: Understand Before You Sign

`Related posts`


SteamVR vient de rendre la VR plus nette. Voici ce que j'y mettrais.
SteamVR vient de rendre la VR plus nette. Voici ce que j'y mettrais. Bonjour, Cette semaine, Valve a publié une mise à j
0dev.to
SQLPad admin and service tokens concentrate database reach
SQLPad admin and service tokens concentrate database reach A SQL console can be more sensitive than the databases it rea
0dev.to
Game Renewals: Implement Recurring User Reminders Using Cron and Queues in SaaS Backend
A gaming SaaS reminder is easy to schedule until the business deadline moves, a worker restarts, or the same renewal not
0dev.to
When Your Vision Therapy Platform Gets a $15B Vote of Confidence — A Builder's Personal Take on Meta's Bond Sale and…
When Your Vision Therapy Platform Gets a $15B Vote of Confidence — A Builder's Personal Take on Meta's Bond Sale and Amb
0dev.to
From SoWa Power Station to the Binding Pocket: Why Structural Biology Needs Its Hallway Moment
From SoWa Power Station to the Binding Pocket: Why Structural Biology Needs Its Hallway Moment I still remember the hum 
0dev.to
Claude Personal Finance: What It Actually Shows You
Anthropic launched Personal Finance in Claude on November 5, a beta that connects bank and card accounts through Plaid I
0dev.to AI