Backpressure in practice: queues, 429s, and client retries that do not stampede

Systems do not get infinite memory when traffic spikes. If you accept every request into an unbounded queue, you trade a quick 429 for a slow death—GC thrash, stale work, and timeouts that all arrive later together. Backpressure is the decision to refuse or delay work early so the system stays within capacity.

Lab: bounded queue

node content/labs/backpressure-in-practice/demo.mjs

A queue with maxDepth=5 receives 12 jobs quickly. Results:

• first 5 accepted
• remaining rejected with queue_full
• worker processes only accepted jobs

That rejection is the feature. Map it to API design:

Internal signal	Client-visible
queue_full	HTTP 429 or 503 with Retry-After
accepted	202/200 and normal processing

Choose a strategy

Strategy	Use when	Risk if misused
Bound queue + reject	Spiky load, durable clients	Clients ignore 429 and stampede
Shed load (drop lowest priority)	Mixed criticality	Dropping critical work
Slow down producers	You control both ends	Needs cooperative clients
Autoscale	Work is parallel and DB allows	Multiplies connection demand

Backpressure without client cooperation just moves the pile to the client. Publish retry rules.

Client retries that do not stampede

Bad:

on any error: retry immediately, 50 times

Better:

retry only on 429/503/network
max attempts: 3–5
exponential backoff + jitter
honor Retry-After
idempotent requests only for non-safe methods

Example delay: min(cap, base 2^attempt) random(0.5, 1.5).

Pair with idempotency keys on writes (see the webhook article) so retries are safe.

Where work piles up

Instrument all of:

• request queue depth
• thread/pool wait
• DB connections
• downstream latency

If you only look at CPU, you miss the queue behind a 200 OK facade (async accepted, never processed).

HTTP status semantics (be consistent)

Teams argue about 429 vs 503. Pick one policy and document it:

Status	Meaning to clients	Typical use
429	You are sending too much; slow down	Per-tenant or global rate / queue full with retry hope
503	Service cannot take work now	Dependency down, shedding under protection
504	We waited on someone else too long	Upstream timeout (not the same as “please retry slower”)

Always prefer a machine-readable body:

{
  "error_code": "queue_full",
  "retry_after_ms": 750
}

and mirror delay in the Retry-After header when you can. Clients that only retry on network errors will still stampede; publish a short client library note or runbook snippet.

Worked numbers from the lab shape

Assume:

• maxDepth = 5
• worker hold = 50 ms per job
• steady arrival = 20 jobs/s

Little’s law says average concurrency ≈ 20 × 0.05 = 1, so depth 5 is plenty until a burst. A 12-job instantaneous burst fills the queue and rejects 7. That is correct if those 7 can retry later; it is a product incident if the UI treats every 429 as a hard failure with no backoff.

Production translation:

1. Set depth from memory and freshness, not from “feels big.”
2. Alert when reject rate rises and depth is pegged—capacity problem—not when a single client misbehaves (rate-limit that client).
3. Separate user-facing queues from async jobs so interactive traffic is not buried behind batch.

Interaction with connection pools

Backpressure that only lives in the HTTP layer can still kill the database: rejected users retry, accepted work still opens pools. Coordinate:

• HTTP reject when queue depth high or pool wait high
• Autoscale limits that respect instances × pool_size (see the connection-pool article)
• Idempotent POSTs so retries after 429 do not double-charge

Validation

• Load test with a hard pass/fail: error budget for 429 vs 5xx (429 can be “correct”)
• Chaos: block downstream; ensure you shed instead of unbounded buffer
• Dashboard: depth + reject rate + p95 latency together
• Client chaos: 100 workers retrying without jitter should not collapse the fleet if server bounds hold

Runbook snippet

1. Alert: queue_depth at max for N minutes or reject_rate high.
2. Check downstream latency and DB pool wait—are you the bottleneck or the victim?
3. If victim: keep rejecting; fix dependency.
4. If you are slow: increase workers only within pool budget, or shed more aggressively.
5. Tell clients/status page: “retry with backoff,” not “spam refresh.”

Summary

Backpressure is capacity made visible. Bound the queue, reject early with clear 429/503 semantics, teach clients to back off with jitter, keep writes idempotent, and align rejects with pool budgets. The lab’s twelve-vs-five example is small; the production version is the same idea with metrics, headers, and SLO language attached.