Most Terraform tutorials show you how to stand up an EC2 instance or a GCS bucket in about 12 lines of code. That part is genuinely easy. What nobody covers is what happens six months later when you have three teams touching the same codebase, four environments that have drifted from each other, a state file that's grown to 40,000 lines, and a terraform plan that takes eight minutes to run.
Terraform infrastructure as code is one of the highest-leverage practices an engineering team can adopt — but only if you structure it deliberately from the start. Retrofitting structure into a flat mess of .tf files is painful. Doing it right upfront is not that much harder.
This post covers the patterns that actually hold up in production environments with multiple engineers and real complexity.
The Module Structure That Scales
The most important decision you'll make in Terraform is how to organize your code. The two failure modes are:
- One giant root module — everything in a single directory,
main.tfwith 2,000 lines, no reuse, impossible to review - Over-modularized too early — modules for everything including individual security group rules, making changes that should take 5 minutes take 45
The right structure for most teams:
infrastructure/
├── modules/ # reusable building blocks
│ ├── vpc/
│ ├── gke-cluster/
│ ├── cloud-sql/
│ └── app-service/
├── environments/
│ ├── prod/
│ │ ├── main.tf # composes modules
│ │ ├── variables.tf
│ │ └── terraform.tfvars
│ ├── staging/
│ └── dev/
└── shared/ # cross-environment resources (DNS, IAM roots)
Modules are your internal building blocks — they encapsulate the details of how a thing is built. A gke-cluster module might accept 8 input variables and manage 15 resources internally. The caller doesn't need to know any of that.
Environments are compositions — they wire together modules with environment-specific values. The prod/main.tf calls ../modules/gke-cluster with production-sized node pools; the dev/main.tf calls the same module with smaller ones. Same structure, different values, no copied code.
The rule for when to extract a module: if you'd configure this resource differently across environments, it belongs in a module with a variable for that difference. If every environment would use the exact same configuration, inline it.
State Management: The Part Nobody Gets Right
Terraform state is where most teams eventually run into serious problems. A few non-negotiable practices:
Remote State With Locking
Local state files do not belong in a collaborative environment. Full stop. Use a remote backend with state locking from day one:
# GCP — backend.tf
terraform {
backend "gcs" {
bucket = "my-org-terraform-state"
prefix = "environments/prod"
}
}
# AWS — backend.tf
terraform {
backend "s3" {
bucket = "my-org-terraform-state"
key = "environments/prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-state-lock"
encrypt = true
}
}
The GCS backend handles locking natively via object versioning. The S3 backend requires a DynamoDB table for locking — provision it before your first terraform init or you'll be cleaning up concurrent state corruption.
One State File Per Environment
Never share state between environments. The blast radius of a bad terraform apply should be contained to one environment. Separate state files enforce that boundary at the infrastructure level, not just by convention.
state bucket/
├── environments/
│ ├── prod/terraform.tfstate
│ ├── staging/terraform.tfstate
│ └── dev/terraform.tfstate
└── shared/terraform.tfstate
Cross-Environment Data with Remote State Data Sources
When one environment needs to reference resources from another (say, a shared VPC ID or a KMS key ARN), use terraform_remote_state rather than hardcoding values:
data "terraform_remote_state" "shared" {
backend = "gcs"
config = {
bucket = "my-org-terraform-state"
prefix = "shared"
}
}
# Now reference outputs from the shared workspace
resource "google_container_cluster" "main" {
network = data.terraform_remote_state.shared.outputs.vpc_id
}
This creates an explicit dependency between state files and makes the relationship visible rather than hidden in a hardcoded string.
Workspace Strategy: What Works and What Doesn't
Terraform workspaces sound like the perfect solution for managing multiple environments. They're often not.
Workspaces work well for: ephemeral environments that are structurally identical (think: spin up a temporary environment per PR, tear it down after merge). Same configuration, different state, different prefix.
Workspaces work poorly for: production vs staging vs dev, where the environments genuinely differ in size, configuration, and risk tolerance. When you start writing terraform.workspace == "prod" ? large_instance : small_instance conditions everywhere, your code becomes hard to reason about and easy to misconfigure.
The directory-per-environment structure above handles the prod/staging/dev case more cleanly — the tradeoff is some duplication in variable declarations, which is acceptable.
Dependency Management and depends_on
One of the most common Terraform bugs in production is race conditions between resources that Terraform thinks are independent but actually depend on each other. The classic example: creating a GKE cluster and immediately running a Helm chart against it before the API server is fully ready.
Explicit depends_on where the dependency isn't captured in a resource reference:
resource "helm_release" "monitoring_stack" {
name = "monitoring"
chart = "kube-prometheus-stack"
repository = "https://prometheus-community.github.io/helm-charts"
namespace = "monitoring"
# The cluster exists in Terraform's graph, but the API server
# might not be ready — make the dependency explicit
depends_on = [
google_container_cluster.main,
google_container_node_pool.primary,
]
}
If you're seeing flaky terraform apply runs that work on retry, missing depends_on is the first thing to check.
Preventing Accidental Destruction
Terraform is very good at doing exactly what you tell it to do. Including destroying production databases when you rename a resource.
Lifecycle Rules
resource "google_sql_database_instance" "main" {
name = "prod-postgres"
database_version = "POSTGRES_15"
lifecycle {
prevent_destroy = true # terraform destroy will error on this resource
}
}
Add prevent_destroy = true to every stateful resource in production: databases, storage buckets, KMS keys, VPCs. Terraform will refuse to destroy them even if you ask it to, requiring you to explicitly remove the lifecycle block first — which is exactly the kind of friction you want before destroying a production database.
Plan Review in CI
Never run terraform apply directly in production from a developer's machine. The workflow:
# .github/workflows/terraform.yml
on:
pull_request:
paths: ['infrastructure/**']
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- name: Plan
run: |
cd infrastructure/environments/prod
terraform init
terraform plan -out=tfplan
- name: Upload plan
uses: actions/upload-artifact@v4
with:
name: tfplan
path: infrastructure/environments/prod/tfplan
The plan output becomes a PR comment (tools like Atlantis and Spacelift do this natively). A human reviews it before merge. The apply happens automatically after merge, using the saved plan — so what was reviewed is exactly what gets applied.
Managing Drift
Infrastructure drift — where the actual state of your cloud resources diverges from what Terraform believes — is inevitable at scale. Someone clicked something in the console, a resource was auto-modified by a cloud provider update, a previous apply partially failed.
Regular terraform plan as a Health Check
Run a plan on each environment on a schedule (daily or weekly) and alert on non-empty output. A non-empty plan in an environment that hasn't been intentionally changed is a signal worth investigating.
# Exit code 2 = plan has changes; exit code 0 = no changes
terraform plan -detailed-exitcode
if [ $? -eq 2 ]; then
echo "Drift detected in $ENVIRONMENT" | send_alert
fi
Import for Resources Created Outside Terraform
When a resource exists in your cloud but not in your Terraform state, terraform import brings it under management:
# Import an existing GCS bucket into state
terraform import google_storage_bucket.logs my-org-logs-bucket
# Import an existing AWS security group
terraform import aws_security_group.app sg-0abc123def456789
After importing, run terraform plan — if it shows no changes, the resource matches your configuration. If it shows changes, your configuration doesn't yet match reality. Reconcile before applying.
Common Pitfalls to Avoid
Putting secrets in .tfvars files. Use a secrets manager and fetch values at runtime, or use environment variables (TF_VAR_db_password). Never commit a .tfvars file with real credentials to version control.
Ignoring terraform fmt and terraform validate in CI. These are free checks that catch syntax errors and obvious type mismatches before a plan even runs. Add them as required status checks.
Not pinning provider versions. A provider update can change behavior in ways that break your configuration. Always pin:
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0" # allow 5.x but not 6.0
}
}
required_version = ">= 1.6"
}
Monolithic root modules with 50+ resources. A terraform apply on a 50-resource root module takes forever and the blast radius of a mistake is huge. If your root module has grown beyond 20–25 resources, it's time to split it.
Where to Go From Here
If you're starting from scratch, resist the urge to structure everything perfectly upfront. Start with the directory-per-environment layout, use remote state from day one, and add module extraction as patterns repeat.
If you're inheriting an existing mess — flat directory, local state, no modules — the safest approach is incremental: migrate to remote state first (this is low-risk and high-payoff), then start extracting modules for the resources you're actively changing.
If your Terraform workflow is slowing your team down — slow plans, unclear review process, repeated drift — reach out. Terraform structure and CI integration is something we help teams get right regularly.
Working on something similar?
We help engineering teams implement the practices covered in this post. First call is free.
Start a conversation →