Secret Management

Every running system needs secrets: the password your service uses to reach Postgres, the API key for Stripe, the private key behind your TLS certificate, the token that lets one microservice call another. A secret is any value that grants access and must stay confidential. If it leaks, an attacker gets the same access your code has.

The old habit was to paste these values straight into the codebase, a .env file, or a Kubernetes manifest. That fails badly. Anyone with read access to the repo sees production credentials, the secret gets copied into CI logs and laptop backups, and rotating it means editing files across dozens of services. The 2016 Uber breach happened exactly this way: AWS keys were committed to a private GitHub repo, attackers found them, and pulled data on 57 million users.

Secret management replaces all of that with a single source of truth. The secret lives encrypted in a dedicated store. Applications fetch it at startup or on demand over an authenticated channel, and it is never written to disk in plaintext. The store keeps a full audit trail and can revoke or rotate a secret without anyone touching code.

A secrets manager keeps everything encrypted at rest using a master key, and that master key is itself protected. HashiCorp Vault, for example, starts sealed: the unseal key is split into several shares using Shamir's Secret Sharing, so no single operator can decrypt the store alone. Cloud managers like AWS Secrets Manager back the master key with a hardware security module through KMS.

To read a secret, a client must first prove who it is. This is the hard part, because the credential you use to authenticate is itself a secret, the so called secret zero problem. Real systems solve it with workload identity that is not a long lived password: an AWS IAM role attached to the instance, a Kubernetes service account token, or a short lived cloud identity token. The manager checks that identity against a policy, then hands back the secret over TLS.

Two features make these tools more than encrypted databases. Rotation lets the manager generate a brand new value on a schedule and update the downstream system, so a leaked secret is only useful for a short window. Dynamic secrets go further: Vault can create a fresh, unique database username and password for each client on request, valid for an hour, then delete it. There is no shared standing credential to steal.

Use a secret manager the moment more than one person or more than one machine needs the same credential, which is almost immediately in any real deployment. It is not optional for production systems handling user data, payments, or anything regulated by SOC 2, PCI DSS, or HIPAA, all of which expect access control and audit logs on credentials.

The cost is operational complexity and a new dependency in your critical path. If your application reads its database password from Vault at startup and Vault is down, your service cannot boot. Teams mitigate this with high availability clusters, caching, and sidecar agents that keep a recent copy in memory. There is also latency: an extra authenticated network round trip on cold start, which is why most apps fetch secrets once and cache them rather than per request.

A lighter approach for small teams is encrypted files committed to git, using SOPS or git-crypt. The secret stays encrypted in the repo and is decrypted at deploy time with a KMS key. This needs no running server, but it gives you no rotation, no audit trail, and no dynamic secrets, so most organizations graduate to a managed service as they scale.

Picture a payments service that needs to read from Postgres and call the Stripe API. Without secret management, both the database password and the Stripe key sit in an environment variable baked into the container image. When an engineer leaves the company, you would have to rebuild and redeploy every service to rotate those values, and most teams simply never do.

With Vault, the container starts with only a Kubernetes service account. It exchanges that token for a Vault token, then requests two things. For Stripe it reads a static secret, and Vault logs the read with a timestamp and the workload identity. For Postgres it requests a dynamic credential, and Vault connects to the database, creates a one hour user, and returns it. The service connects, and an hour later that database user no longer exists.

Now rotation is a config change, not a redeploy. If the Stripe key ever leaks, you revoke and replace it in one place and every consumer picks up the new value on its next fetch. The audit log tells your security team exactly which service read which secret and when, which is the evidence an auditor asks for.