Skip to content

Secrets in Deployments

There's a graveyard of hacked startups that leaked API keys in git history — once a secret is committed , you can't uncommit it. The history is forever , git bisect won't save you , and tools like trufflehog crawl public repos specifically looking for your mistakes The rule: if a secret touches your filesystem outside a secrets manager , assume it's compromised

The Secret Sprawl Problem

Secrets end up everywhere: * .env files committed to git * Hardcoded in source code * In CI/CD pipeline logs * In Docker image layers * In Slack messages and emails * In error reporting tools (stack traces with env vars) * In artifact files (compiled configs , build output)

The cost of leaked secrets: * Database exposed — data breach , regulatory fines (GDPR , HIPAA , PCI-DSS) * Cloud provider access — crypto mining , data exfiltration , infrastructure takeover * API keys — service abuse , unexpected billing , rate limiting your legit traffic

Secrets Managers — The Right Way

HashiCorp Vault (self-hosted):

# Start Vault in dev mode (NOT for production)
vault server -dev

# Set environment variable
export VAULT_ADDR='http://127.0.0.1:8200'

# Write a secret
vault kv put secret/myapp \
  DATABASE_URL=postgres://user:pass@prod-db:5432/MyApp \
  JWT_SECRET=supersecretkey123

# Read a secret
vault kv get secret/myapp

# From Node.js
npm install @hashicorp/vault-client

const vault = require('@hashicorp/vault-client')

const client = new vault.Client({
  address: process.env.VAULT_ADDR,
  token: process.env.VAULT_TOKEN
})

async function getConfig() {
  const response = await client.secrets.kv.v2.read({
    mount: 'secret',
    path: 'myapp'
  })
  return response.data.data
}

AWS Secrets Manager (managed):

# Create secret
aws secretsmanager create-secret \
  --name myapp/production \
  --secret-string '{"DATABASE_URL":"postgres://...","JWT_SECRET":"..."}'

# Retrieve in Node.js
npm install @aws-sdk/client-secrets-manager

const { SecretsManagerClient, GetSecretValueCommand } = require('@aws-sdk/client-secrets-manager')

const client = new SecretsManagerClient({ region: 'us-east-1' })

async function getSecret() {
  const response = await client.send(
    new GetSecretValueCommand({ SecretId: 'myapp/production' })
  )
  return JSON.parse(response.SecretString)
}

Doppler (cloud-native): Doppler is the middle ground — simpler than Vault , more powerful than .env files

# Fetch secrets as env vars
doppler secrets download --format env --config prd > .env.production

# Or inject directly
doppler run -- node server.js

# Works in CI with GitHub Actions
doppler run --command="npm run deploy" --token=$DOPPLER_TOKEN

Encrypted .env Files — Better Than Plaintext

# Install git-crypt
sudo apt install git-crypt

# Initialize in your repo
git-crypt init

# Specify which files to encrypt in .gitattributes
echo ".env filter=git-crypt diff=git-crypt" >> .gitattributes

# Add GPG key or symmetric key
git-crypt add-gpg-user KEY_ID

# Lock/unlock
git-crypt lock
git-crypt unlock

# The .env file is encrypted in git , decrypted on checkout
# Alternative: sops (SOPS) from Mozilla
# Encrypts values but keeps structure visible
# .sops.yaml
creation_rules:
  - pgp: >-
    YOUR_PGP_KEY_FINGERPRINT
# .env.encrypted — values encrypted , keys visible
DATABASE_URL=ENC[AES256_GCM,data:encryptedbase64==,iv:...,tag:...]
JWT_SECRET=ENC[AES256_GCM,data:...,iv:...,tag:...]

Avoiding Secrets in Git History — Prevention Is Cheaper Than Cleanup

Once a secret is in git , it's in every clone forever. Even force-pushing doesn't remove it from forks or local clones. Preventive measures:

# git-secrets — scan before commit
git secrets --scan
git secrets --add 'DATABASE_URL=postgres://.*:.*@'

# Install as pre-commit hook
git secrets --install

# trufflehog — scan entire repo history
trufflehog git file://. --since-commit HEAD~10

# detect-secrets from Yelp
detect-secrets scan . > .secrets.baseline
detect-secrets audit .secrets.baseline

# GitHub secret scanning (enabled by default on public repos)

Pre-commit hook that blocks secrets:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.5.0
    hooks:
      - id: detect-private-key

  - repo: https://github.com/Yelp/detect-secrets
    rev: v1.4.0
    hooks:
      - id: detect-secrets
        args: ['--baseline', '.secrets.baseline']

  - repo: https://github.com/awslabs/git-secrets
    rev: master
    hooks:
      - id: git-secrets

When you accidentally commit a secret (the damage control script):

# THIS DOES NOT REMOVE FROM FORKS OR CLONES
# But it's better than nothing

# 1. Find the commit
git log --all --oneline | grep -i "secret\|password\|key"

# 2. Remove with BFG Repo-Cleaner
java -jar bfg.jar --replace-text passwords.txt repo.git

# 3. Force push (requires force push override)
git push --force

# 4. Rotate the secret IMMEDIATELY
# The old secret is still compromised

# 5. Remove from GitHub cache
# Contact GitHub support to purge cached data

Secrets in Docker — Build Args vs Runtime Env

Build-time secrets (don't end up in image layers):

# Dockerfile
RUN --mount=type=secret,id=npm_token \
    NPM_TOKEN=$(cat /run/secrets/npm_token) \
    npm install --registry=https://private-registry.com

# Build command
docker build --secret id=npm_token,src=./npm_token.txt -t myapp .

Runtime secrets (injected when container starts):

services:
  app:
    image: myapp
    environment:
      # These are passed at runtime , not baked into the image
      - DATABASE_URL=${DATABASE_URL}
      - JWT_SECRET=${JWT_SECRET}
    secrets:
      - db_password

secrets:
  db_password:
    file: ./secrets/db_password.txt

The fatal mistake — ENV in Dockerfile:

# NEVER — this bakes the secret into the image
ENV DATABASE_URL=postgres://user:pass@prod:5432/myapp
# Anyone who can pull the image can extract this

Secret Rotation — Have a Plan Before You Need One

# Manual rotation
doppler secrets set DATABASE_URL="postgres://newuser:newpass@newhost:5432/MyApp"
doppler secrets set --config prd DATABASE_URL="postgres://..."
doppler download --format env --config prd > .env.production

# Automated rotation (AWS Secrets Manager)
aws secretsmanager rotate-secret --secret-id myapp/production
# Lambda function handles the actual credential change in the database

# Zero-downtime rotation pattern:
# 1. Deploy app that accepts BOTH old and new credentials
# 2. Rotate the secret
# 3. Deploy app that only accepts new credentials
# 4. Remove old credential

next → devops_09_scaling.md