We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
This idea carries a full system design question on its own. Each walks through the full answer.
System Design Masterclass · Security Architecture
Free preview: the first 2 parts of this 7-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design courseread the rest of this lesson in
System Design Masterclass
Unlock this lesson and the 751 other paid lessons
$25 once, by PayPal
See the System Design coursein Security Architecture
This lesson is part of the full System Design Masterclass. Unlock it along with 769 other lessons for a one-time payment.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime Access18 Foundation lessons are free, no payment needed. Try them now
It is easy and good
“It is really nice and good Please join the course”
A developer pushed code to a public GitHub repository. The commit included an .env file with AWS access keys. Within minutes, literally minutes, a bot scanning GitHub for leaked credentials found the keys, spun up dozens of high-spec EC2 instances across multiple regions, and started mining cryptocurrency. By the time the developer noticed and revoked the keys four hours later, the bill was $14,000.
This is not rare. GitHub's secret scanning service detected over 10 million leaked secrets in public repositories in 2023. Trufflehog, a popular secret detection tool, regularly finds valid AWS keys, Stripe keys, database connection strings, and private certificates in public repos. Bots scrape GitHub's real-time event stream and can exploit a leaked credential within 15 seconds of the push.
The problem extends far beyond Git. Secrets leak through logs (many CI tools log environment variables by default), Docker image layers (docker history reveals every layer including deleted secrets), Slack messages (searchable, backed up, no expiry), Jupyter notebooks committed to repos, and state files stored in S3 without encryption.
Where secrets should NOT live:
| Bad Practice | Why It Is Dangerous |
|---|---|
| Hardcoded in source code | In Git history permanently. Visible to every developer. Embedded in build artifacts |
.env files in repos | Even .gitignore fails if committed once. Git history is permanent, git filter-branch does not help if others have cloned |
| logs | GitHub Actions, Jenkins, and CircleCI have all had incidents where secrets were printed to build logs |
| image layers | Every COPY and ENV instruction creates a layer. docker history --no-trunc reveals them all, including "deleted" files |
| Slack or email | Searchable, replicated, backed up, no access control, no expiration |
| state | Contains every resource attribute in plain text, including database passwords and API keys |
Where secrets SHOULD live:
| Solution | How It Works | Rotation Support | Cost |
|---|---|---|---|
| AWS Secrets Manager | Encrypted with KMS, IAM access control, Lambda rotation | Built-in auto-rotation for RDS, Redshift, DocumentDB | $0.40/secret/month + $0.05 per 10K API calls |
| GCP Secret Manager | Encrypted with Cloud KMS, IAM, versioning | Via Cloud Functions | $0.06/secret version/month |
| HashiCorp Vault | Self-hosted or HCP, dynamic secrets, transit engine | Dynamic secrets with (minutes to hours) | Open source or $0.03/hr (HCP) |
| Doppler | SaaS, syncs to all clouds, CLI integration | Manual or webhook-triggered | Free tier up to 5 users |
| Secrets | Base64-encoded in etcd, encrypted at rest with KMS | Manual (use External Secrets Operator for sync) |
Dynamic secrets are the gold standard. Instead of a static database password that lives forever, Vault generates a unique database credential for each application instance, valid for 1 hour. When it expires, the application requests a new one. A leaked dynamic secret is useless after the TTL expires. Compare that to a static password that was valid for 3 years and shared across 15 services.
| Free (part of K8s) |