Rotating secrets without downtime: dual-read, dual-write, and cutover

Secret rotation fails when only one value is valid at a time and instances roll slowly. The safe pattern is overlap: accept old and new during a window, then drop old. This article shows that dual-read pattern with a tiny lab and a deploy order you can reuse for API keys, HMAC secrets, or DB passwords (with extra care).

Lab

node content/labs/secrets-rotation-no-downtime/demo.mjs

Phases:

1. Only secret_v1 verifies
2. After rotate: both v1 and v2 verify
3. After drop previous: only v2

That is the whole production idea, minus durable secret storage.

Blast radius first

Secret	Blast radius	Notes
JWT signing key	all sessions	need dual verify or short drain
Webhook HMAC	provider callbacks	dual keys in provider + app
DB password	all app instances	often needs pool recycle + dual user
Third-party API key	one integration	easier; still overlap

Name owners and systems before rotating anything.

Generic dual-read sequence

1. Generate new secret in vault/KMS (do not paste chat).
2. Install new as secondary everywhere that verifies (app config previous/primary).
3. Deploy dual-read code if not already present.
4. Switch primary to new for signing/outbound calls.
5. Observe error rates (auth failures).
6. Remove old after the maximum token/request age passes.
7. Revoke old at the provider side last when applicable.

Deploy order matters

• Verifiers first, issuers second for signing keys: old tokens must still verify while new tokens start issuing.
• For outbound API keys, put the new key in the provider allowlist before apps send it.

DB password special case

Common approach:

1. Create new DB user/password (or alter with dual users)
2. App dual-config is harder; often: add new credential, rolling restart, then drop old user
3. Ensure pool does not keep dead connections forever—recycle on auth errors

Do not rotate the only superuser in-place without a rescue session.

Failure cases

Failure	Symptom	Mitigation
Dropped old too early	spike 401	re-enable previous temporarily
Only half of instances updated	intermittent auth	slower roll, health checks
Provider not updated	outbound 401	order checklist
Logged secrets	security incident	vault + redaction

Verification

• Synthetic probe: obtain token with new key, verify on all pods
• Metric: auth failure rate by version tag if available
• Canary tenant before global

JWT / session signing example

Overlap window length should be at least:

max_token_lifetime + deploy_time + safety_margin

Sequence:

1. Deploy verifiers that accept kid (key id) for both keys.
2. Start issuing tokens with kid=v2.
3. Wait until all kid=v1 tokens expire (or force re-login if you accept that).
4. Remove v1 from the verify set.
5. Delete v1 material from the vault.

If you only swap a single global secret with no kid, you force a hard cut—every in-flight token dies. That is a product decision, not a rotation best practice.

Webhook HMAC example

1. Generate secret_v2 in your app vault.
2. Configure the provider with both secrets if it supports dual secrets; otherwise schedule a short dual-verify in your app while the provider still sends v1.
3. Switch provider to sign with v2.
4. Observe signature failures (should stay flat).
5. Drop v1 after the provider’s retry window.

Never log raw signatures or secrets when debugging “invalid signature”—log kid, timestamp skew, and body hash only.

Checklist card for on-call

Step	Owner	Done
New secret created in vault	security/platform
Dual-read deployed	app
Issuer/provider switched	app
Auth error burn rate normal	on-call
Old secret revoked	security/platform
Runbook updated with key ids	app

Rollback

If auth errors spike after dropping old:

1. Re-enable previous immediately (config/flag).
2. Stop issuing with the suspect new key if errors are on issue path.
3. Leave dual-read up while you diagnose clock skew, wrong key distribution, or partial deploys.
4. Postmortem: which phase skipped verification?

Summary

Rotation without downtime is overlap + order + verification. The lab’s three phases are the checklist: single old → dual → single new. Use key ids for JWTs, dual secrets for webhooks, careful dual users for databases, and a written rollback. Automate storage and access; keep the phase machine human-readable for on-call.