Secrets in Deployments¶
There's a graveyard of hacked startups that leaked API keys in git history — once a secret is committed , you can't uncommit it. The history is forever , git bisect won't save you , and tools like trufflehog crawl public repos specifically looking for your mistakes The rule: if a secret touches your filesystem outside a secrets manager , assume it's compromised
The Secret Sprawl Problem¶
Secrets end up everywhere: * .env files committed to git * Hardcoded in source code * In CI/CD pipeline logs * In Docker image layers * In Slack messages and emails * In error reporting tools (stack traces with env vars) * In artifact files (compiled configs , build output)
The cost of leaked secrets: * Database exposed — data breach , regulatory fines (GDPR , HIPAA , PCI-DSS) * Cloud provider access — crypto mining , data exfiltration , infrastructure takeover * API keys — service abuse , unexpected billing , rate limiting your legit traffic
Secrets Managers — The Right Way¶
HashiCorp Vault (self-hosted):
# Start Vault in dev mode (NOT for production)
vault server -dev
# Set environment variable
export VAULT_ADDR='http://127.0.0.1:8200'
# Write a secret
vault kv put secret/myapp \
DATABASE_URL=postgres://user:pass@prod-db:5432/MyApp \
JWT_SECRET=supersecretkey123
# Read a secret
vault kv get secret/myapp
# From Node.js
npm install @hashicorp/vault-client
const vault = require('@hashicorp/vault-client')
const client = new vault.Client({
address: process.env.VAULT_ADDR,
token: process.env.VAULT_TOKEN
})
async function getConfig() {
const response = await client.secrets.kv.v2.read({
mount: 'secret',
path: 'myapp'
})
return response.data.data
}
AWS Secrets Manager (managed):
# Create secret
aws secretsmanager create-secret \
--name myapp/production \
--secret-string '{"DATABASE_URL":"postgres://...","JWT_SECRET":"..."}'
# Retrieve in Node.js
npm install @aws-sdk/client-secrets-manager
const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager')
const client = new SecretsManagerClient({ region: 'us-east-1' })
async function getSecret() {
const response = await client.send(
new GetSecretValueCommand({ SecretId: 'myapp/production' })
)
return JSON.parse(response.SecretString)
}
Doppler (cloud-native): Doppler is the middle ground — simpler than Vault , more powerful than .env files
# Fetch secrets as env vars
doppler secrets download --format env --config prd > .env.production
# Or inject directly
doppler run -- node server.js
# Works in CI with GitHub Actions
doppler run --command="npm run deploy" --token=$DOPPLER_TOKEN
Encrypted .env Files — Better Than Plaintext¶
# Install git-crypt
sudo apt install git-crypt
# Initialize in your repo
git-crypt init
# Specify which files to encrypt in .gitattributes
echo ".env filter=git-crypt diff=git-crypt" >> .gitattributes
# Add GPG key or symmetric key
git-crypt add-gpg-user KEY_ID
# Lock/unlock
git-crypt lock
git-crypt unlock
# The .env file is encrypted in git , decrypted on checkout
# Alternative: sops (SOPS) from Mozilla
# Encrypts values but keeps structure visible
# .sops.yaml
creation_rules:
- pgp: >-
YOUR_PGP_KEY_FINGERPRINT
# .env.encrypted — values encrypted , keys visible
DATABASE_URL=ENC[AES256_GCM,data:encryptedbase64==,iv:...,tag:...]
JWT_SECRET=ENC[AES256_GCM,data:...,iv:...,tag:...]
Avoiding Secrets in Git History — Prevention Is Cheaper Than Cleanup¶
Once a secret is in git , it's in every clone forever. Even force-pushing doesn't remove it from forks or local clones. Preventive measures:
# git-secrets — scan before commit
git secrets --scan
git secrets --add 'DATABASE_URL=postgres://.*:.*@'
# Install as pre-commit hook
git secrets --install
# trufflehog — scan entire repo history
trufflehog git file://. --since-commit HEAD~10
# detect-secrets from Yelp
detect-secrets scan . > .secrets.baseline
detect-secrets audit .secrets.baseline
# GitHub secret scanning (enabled by default on public repos)
Pre-commit hook that blocks secrets:
# .pre-commit-config.yaml
repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v4.5.0
hooks:
- id: detect-private-key
- repo: https://github.com/Yelp/detect-secrets
rev: v1.4.0
hooks:
- id: detect-secrets
args: ['--baseline', '.secrets.baseline']
- repo: https://github.com/awslabs/git-secrets
rev: master
hooks:
- id: git-secrets
When you accidentally commit a secret (the damage control script):
# THIS DOES NOT REMOVE FROM FORKS OR CLONES
# But it's better than nothing
# 1. Find the commit
git log --all --oneline | grep -i "secret\|password\|key"
# 2. Remove with BFG Repo-Cleaner
java -jar bfg.jar --replace-text passwords.txt repo.git
# 3. Force push (requires force push override)
git push --force
# 4. Rotate the secret IMMEDIATELY
# The old secret is still compromised
# 5. Remove from GitHub cache
# Contact GitHub support to purge cached data
Secrets in Docker — Build Args vs Runtime Env¶
Build-time secrets (don't end up in image layers):
# Dockerfile
RUN --mount=type=secret,id=npm_token \
NPM_TOKEN=$(cat /run/secrets/npm_token) \
npm install --registry=https://private-registry.com
# Build command
docker build --secret id=npm_token,src=./npm_token.txt -t myapp .
Runtime secrets (injected when container starts):
services:
app:
image: myapp
environment:
# These are passed at runtime , not baked into the image
- DATABASE_URL=${DATABASE_URL}
- JWT_SECRET=${JWT_SECRET}
secrets:
- db_password
secrets:
db_password:
file: ./secrets/db_password.txt
The fatal mistake — ENV in Dockerfile:
# NEVER — this bakes the secret into the image
ENV DATABASE_URL=postgres://user:pass@prod:5432/myapp
# Anyone who can pull the image can extract this
Secret Rotation — Have a Plan Before You Need One¶
# Manual rotation
doppler secrets set DATABASE_URL="postgres://newuser:newpass@newhost:5432/MyApp"
doppler secrets set --config prd DATABASE_URL="postgres://..."
doppler download --format env --config prd > .env.production
# Automated rotation (AWS Secrets Manager)
aws secretsmanager rotate-secret --secret-id myapp/production
# Lambda function handles the actual credential change in the database
# Zero-downtime rotation pattern:
# 1. Deploy app that accepts BOTH old and new credentials
# 2. Rotate the secret
# 3. Deploy app that only accepts new credentials
# 4. Remove old credential
next → devops_09_scaling.md