Most Terraform tutorials show you how to stand up an EC2 instance or a GCS bucket in about 12 lines of code. That part is genuinely easy. What nobody covers is what happens six months later when you have three teams touching the same codebase, four environments that have drifted from each other, a state file that's grown to 40,000 lines, and a terraform plan that takes eight minutes to run.
Terraform infrastructure as code is one of the highest-leverage practices an engineering team can adopt — but only if you structure it deliberately from the start. Retrofitting structure into a flat mess of .tf files is painful. Doing it right upfront is not that much harder.
This post covers the patterns that actually hold up in production environments with multiple engineers and real complexity.
The most important decision you'll make in Terraform is how to organize your code. The two failure modes are:
main.tf with 2,000 lines, no reuse, impossible to reviewThe right structure for most teams:
infrastructure/
├── modules/ # reusable building blocks
│ ├── vpc/
│ ├── gke-cluster/
│ ├── cloud-sql/
│ └── app-service/
├── environments/
│ ├── prod/
│ │ ├── main.tf # composes modules
│ │ ├── variables.tf
│ │ └── terraform.tfvars
│ ├── staging/
│ └── dev/
└── shared/ # cross-environment resources (DNS, IAM roots)
Modules are your internal building blocks — they encapsulate the details of how a thing is built. A gke-cluster module might accept 8 input variables and manage 15 resources internally. The caller doesn't need to know any of that.
Environments are compositions — they wire together modules with environment-specific values. The prod/main.tf calls ../modules/gke-cluster with production-sized node pools; the dev/main.tf calls the same module with smaller ones. Same structure, different values, no copied code.
The rule for when to extract a module: if you'd configure this resource differently across environments, it belongs in a module with a variable for that difference. If every environment would use the exact same configuration, inline it.
Terraform state is where most teams eventually run into serious problems. A few non-negotiable practices:
Local state files do not belong in a collaborative environment. Full stop. Use a remote backend with state locking from day one:
# GCP — backend.tf
terraform {
backend "gcs" {
bucket = "my-org-terraform-state"
prefix = "environments/prod"
}
}
# AWS — backend.tf
terraform {
backend "s3" {
bucket = "my-org-terraform-state"
key = "environments/prod/terraform.tfstate"
region = "us-east-1"
dynamodb_table = "terraform-state-lock"
encrypt = true
}
}
The GCS backend handles locking natively via object versioning. The S3 backend requires a DynamoDB table for locking — provision it before your first terraform init or you'll be cleaning up concurrent state corruption.
Never share state between environments. The blast radius of a bad terraform apply should be contained to one environment. Separate state files enforce that boundary at the infrastructure level, not just by convention.
state bucket/
├── environments/
│ ├── prod/terraform.tfstate
│ ├── staging/terraform.tfstate
│ └── dev/terraform.tfstate
└── shared/terraform.tfstate
When one environment needs to reference resources from another (say, a shared VPC ID or a KMS key ARN), use terraform_remote_state rather than hardcoding values:
data "terraform_remote_state" "shared" {
backend = "gcs"
config = {
bucket = "my-org-terraform-state"
prefix = "shared"
}
}
# Now reference outputs from the shared workspace
resource "google_container_cluster" "main" {
network = data.terraform_remote_state.shared.outputs.vpc_id
}
This creates an explicit dependency between state files and makes the relationship visible rather than hidden in a hardcoded string.
Terraform workspaces sound like the perfect solution for managing multiple environments. They're often not.
Workspaces work well for: ephemeral environments that are structurally identical (think: spin up a temporary environment per PR, tear it down after merge). Same configuration, different state, different prefix.
Workspaces work poorly for: production vs staging vs dev, where the environments genuinely differ in size, configuration, and risk tolerance. When you start writing terraform.workspace == "prod" ? large_instance : small_instance conditions everywhere, your code becomes hard to reason about and easy to misconfigure.
The directory-per-environment structure above handles the prod/staging/dev case more cleanly — the tradeoff is some duplication in variable declarations, which is acceptable.
depends_onOne of the most common Terraform bugs in production is race conditions between resources that Terraform thinks are independent but actually depend on each other. The classic example: creating a GKE cluster and immediately running a Helm chart against it before the API server is fully ready.
Explicit depends_on where the dependency isn't captured in a resource reference:
resource "helm_release" "monitoring_stack" {
name = "monitoring"
chart = "kube-prometheus-stack"
repository = "https://prometheus-community.github.io/helm-charts"
namespace = "monitoring"
# The cluster exists in Terraform's graph, but the API server
# might not be ready — make the dependency explicit
depends_on = [
google_container_cluster.main,
google_container_node_pool.primary,
]
}
If you're seeing flaky terraform apply runs that work on retry, missing depends_on is the first thing to check.
Terraform is very good at doing exactly what you tell it to do. Including destroying production databases when you rename a resource.
resource "google_sql_database_instance" "main" {
name = "prod-postgres"
database_version = "POSTGRES_15"
lifecycle {
prevent_destroy = true # terraform destroy will error on this resource
}
}
Add prevent_destroy = true to every stateful resource in production: databases, storage buckets, KMS keys, VPCs. Terraform will refuse to destroy them even if you ask it to, requiring you to explicitly remove the lifecycle block first — which is exactly the kind of friction you want before destroying a production database.
Never run terraform apply directly in production from a developer's machine. The workflow:
# .github/workflows/terraform.yml
on:
pull_request:
paths: ['infrastructure/**']
jobs:
plan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: hashicorp/setup-terraform@v3
- name: Plan
run: |
cd infrastructure/environments/prod
terraform init
terraform plan -out=tfplan
- name: Upload plan
uses: actions/upload-artifact@v4
with:
name: tfplan
path: infrastructure/environments/prod/tfplan
The plan output becomes a PR comment (tools like Atlantis and Spacelift do this natively). A human reviews it before merge. The apply happens automatically after merge, using the saved plan — so what was reviewed is exactly what gets applied.
Infrastructure drift — where the actual state of your cloud resources diverges from what Terraform believes — is inevitable at scale. Someone clicked something in the console, a resource was auto-modified by a cloud provider update, a previous apply partially failed.
terraform plan as a Health CheckRun a plan on each environment on a schedule (daily or weekly) and alert on non-empty output. A non-empty plan in an environment that hasn't been intentionally changed is a signal worth investigating.
# Exit code 2 = plan has changes; exit code 0 = no changes
terraform plan -detailed-exitcode
if [ $? -eq 2 ]; then
echo "Drift detected in $ENVIRONMENT" | send_alert
fi
When a resource exists in your cloud but not in your Terraform state, terraform import brings it under management:
# Import an existing GCS bucket into state
terraform import google_storage_bucket.logs my-org-logs-bucket
# Import an existing AWS security group
terraform import aws_security_group.app sg-0abc123def456789
After importing, run terraform plan — if it shows no changes, the resource matches your configuration. If it shows changes, your configuration doesn't yet match reality. Reconcile before applying.
Putting secrets in .tfvars files. Use a secrets manager and fetch values at runtime, or use environment variables (TF_VAR_db_password). Never commit a .tfvars file with real credentials to version control.
Ignoring terraform fmt and terraform validate in CI. These are free checks that catch syntax errors and obvious type mismatches before a plan even runs. Add them as required status checks.
Not pinning provider versions. A provider update can change behavior in ways that break your configuration. Always pin:
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "~> 5.0" # allow 5.x but not 6.0
}
}
required_version = ">= 1.6"
}
Monolithic root modules with 50+ resources. A terraform apply on a 50-resource root module takes forever and the blast radius of a mistake is huge. If your root module has grown beyond 20–25 resources, it's time to split it.
If you're starting from scratch, resist the urge to structure everything perfectly upfront. Start with the directory-per-environment layout, use remote state from day one, and add module extraction as patterns repeat.
If you're inheriting an existing mess — flat directory, local state, no modules — the safest approach is incremental: migrate to remote state first (this is low-risk and high-payoff), then start extracting modules for the resources you're actively changing.
If your Terraform workflow is slowing your team down — slow plans, unclear review process, repeated drift — reach out. Terraform structure and CI integration is something we help teams get right regularly.