Secrets Management in Production: Vault, Cloud Secret Managers, and the External Secrets Operator

Somewhere in your organization right now, there's a .env file with a production database password sitting in a Slack DM, a personal laptop, or — worse — a git history that "we cleaned up" three years ago but never actually rewrote. Secrets management is the practice of making that scenario structurally impossible instead of hoping nobody notices.

If your current secrets strategy is "environment variables set in the CI dashboard and a shared 1Password vault," you don't have secrets management. You have secrets storage. The difference matters the day a credential leaks and you need to know exactly what it could access, who touched it, and how to rotate it without a deploy freeze.

What secrets management actually means

Secrets management is four capabilities working together, not one tool:

Centralized storage — one system of record for credentials, not scattered across CI variables, config files, and password managers.
Access control — least-privilege policies that scope which identity can read which secret, enforced by the system, not by convention.
Rotation — the ability to invalidate and replace a credential without a code deploy, ideally on a schedule and always on-demand.
Audit trail — a log of who accessed what secret and when, so "was this credential compromised" has an answer instead of a guess.

Static secrets — a password that's valid until someone remembers to change it — fail all four by default. The industry's real trajectory is toward dynamic secrets: short-lived, auto-expiring credentials minted on demand and useless an hour later even if they leak. That shift is the throughline for everything below.

The three real options

HashiCorp Vault

Vault is the general-purpose answer. It does static KV storage, but its actual value is dynamic secrets: Vault can generate a Postgres user with a 15-minute TTL, hand it to your application, and revoke it automatically. No human ever knows the long-lived root credential exists.

# Enable the database secrets engine
vault secrets enable database

vault write database/config/prod-postgres \
  plugin_name=postgresql-database-plugin \
  connection_url="postgresql://{{username}}:{{password}}@db.prod.internal:5432/app" \
  allowed_roles="readonly" \
  username="vault-admin" \
  password="$VAULT_DB_ADMIN_PW"

vault write database/roles/readonly \
  db_name=prod-postgres \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' \
    VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
  default_ttl="15m" \
  max_ttl="1h"

# An app fetches a fresh, scoped credential at runtime
vault read database/creds/readonly

Vault is the right call when you need dynamic secrets across heterogeneous systems (databases, cloud IAM, PKI, SSH) and you're willing to run and operate another piece of infrastructure. That operational cost is real — unsealing, HA setup, storage backend choice — and it's the main reason teams reach for it later than they should.

Cloud-native secret managers

AWS Secrets Manager, GCP Secret Manager, and Azure Key Vault do less than Vault but require zero operational overhead — they're managed services that integrate directly with your cloud's IAM.

aws secretsmanager create-secret \
  --name prod/api/stripe-key \
  --secret-string '{"key":"sk_live_..."}' \
  --tags Key=team,Value=payments

aws secretsmanager rotate-secret \
  --secret-id prod/api/stripe-key \
  --rotation-lambda-arn arn:aws:lambda:us-east-1:123456789:function:rotate-stripe-key \
  --rotation-rules AutomaticallyAfterDays=30

Access control is just IAM policy — no separate auth system to run:

{
  "Effect": "Allow",
  "Action": "secretsmanager:GetSecretValue",
  "Resource": "arn:aws:secretsmanager:us-east-1:123456789:secret:prod/api/stripe-key-*",
  "Condition": {
    "StringEquals": { "aws:PrincipalTag/service": "checkout-api" }
  }
}

If your infrastructure lives entirely in one cloud, start here. You give up cross-platform dynamic secrets (no auto-generated Postgres creds without wiring your own Lambda rotation), but you also give up running a stateful cluster whose sole job is holding your credentials.

Kubernetes-native: SOPS and Sealed Secrets

If your deploy story is GitOps, you may not want a secrets system at all — you want encrypted secrets that live safely in the same git repo as everything else. Mozilla SOPS encrypts values in a YAML/JSON file using a KMS key; Sealed Secrets encrypts against a controller-held keypair so only your cluster can decrypt.

sops --encrypt --kms arn:aws:kms:us-east-1:123456789:key/abcd1234 \
  secrets.yaml > secrets.enc.yaml

git add secrets.enc.yaml && git commit -m "add checkout-api db creds"

This is the lightest-weight option and pairs naturally with a GitOps pipeline — see our take on GitOps beyond CI/CD for the broader pattern. Its weakness is that these are still fundamentally static secrets; you get encryption-at-rest-in-git, not rotation or dynamic issuance.

Wiring it into Kubernetes: External Secrets Operator

Most teams don't pick exactly one of the above — they run Vault or a cloud secret manager as the source of truth and sync into Kubernetes with the External Secrets Operator (ESO). This keeps secrets out of your Kubernetes manifests entirely while still letting pods consume them as native Secret objects.

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: checkout
spec:
  provider:
    vault:
      server: "https://vault.internal:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "checkout-api"
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: checkout-db-creds
  namespace: checkout
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: checkout-db-creds
  data:
    - secretKey: password
      remoteRef:
        key: database/creds/readonly
        property: password

The refreshInterval is the whole point: ESO polls the backend and updates the Kubernetes Secret automatically, so a rotated credential propagates without a redeploy. Your pods mount the secret as a volume or env var exactly like any other Kubernetes secret; they never talk to Vault or AWS directly, and the Vault Kubernetes auth method means no static Vault token is baked into a pod spec either — the pod's service account identity is the credential.

Common pitfalls

Secrets leak through logs and CI output, not just git. A set -x in a shell script, an env | sort for debugging, an exception stack trace that includes a connection string — these are more common leak vectors than committed files. Configure your CI system to mask known secret values in logs, and audit application error handlers for accidental credential inclusion.

Long-lived service account keys never get rotated because nothing forces it. A static AWS access key or a Vault token with no TTL will sit valid for years if nobody's watching. Set max_ttl on every Vault token and lease. For cloud IAM keys you can't avoid, set calendar reminders or, better, use workload identity federation (IAM roles for service accounts, GCP Workload Identity) to eliminate long-lived keys entirely.

Over-permissioned service accounts turn one leak into a full breach. A checkout-api service that can read every secret in Vault, not just its own, means a single compromised pod exposes everything. Scope policies per-application, not per-team — the extra YAML is cheap compared to an incident review.

Secrets in environment variables are visible to anything that can read /proc/<pid>/environ — including a compromised sidecar or a debugging tool a teammate runs in production. Where the tool supports it, prefer mounted files over env vars; a file can be permissioned and isn't dumped by generic process inspection the way env vars are.

Nobody tests the rotation path until the day they need it. A rotation Lambda that's never been triggered outside of --dry-run is a rotation Lambda that will fail during an actual incident. Exercise rotation on a schedule — even a boring 90-day rotation on a secret that "never needs it" — so the mechanism is proven before it's load-bearing.

Where to start

If you're on a single cloud with straightforward Kubernetes deploys: cloud-native secret manager plus External Secrets Operator gets you 90% of the value with none of the Vault operational tax. If you need dynamic database credentials, PKI issuance, or you're multi-cloud, Vault earns its keep. Either way, the migration off .env files and CI-variable sprawl is the actual hard part — the tooling is the easy part.

If your team is still passing production credentials through Slack and hoping, we help engineering orgs design and implement secrets management that survives an audit and an incident review. Get in touch and we'll walk through what fits your stack.