Load testing the right question: capacity, soak, and the failure you care about

“We hit 10k RPS in staging” is not a result. It is a party trick if the question was never written down. Useful load testing starts with a decision: ship, change pool size, buy capacity, or stop a release. This article frames one question, sets criteria first, then uses a tiny simulator to show pass vs fail runs.

Bad load tests you have seen

• Max RPS vanity without latency or error constraints
• Empty cache + tiny DB that cannot miss-storm
• No production-like connection pool settings
• Success defined as “tool finished”
• One-minute spikes only (no soak for leaks)

Write the question

Example:

Can POST /v1/orders/{id}/pay sustain 200 RPS for 15 minutes with p95 under 300 ms and error rate at most 1% when pool_size=11 and max replicas=12?

If the answer is no, you change config or refuse the marketing campaign—not “tune the load generator.”

Criteria before traffic

node content/labs/load-testing-right-question/demo.mjs

The lab encodes criteria:

{ rps: 200, p95Ms: 300, maxErrorRate: 0.01 }

and prints two synthetic runs:

• baseline_pool_ok → pass
• pool_starved → fail (p95 and errors blow the budget)

Production tools (k6, Gatling, vegeta) should print the same kind of boolean against recorded SLOs.

Traffic model

Include:

• Read/write mix matching prod
• Think time or not (be explicit)
• Auth cost
• Dependency stubs vs real (label the difference)
• Warm vs cold cache (see caching article)

Document what is fake. A pass against mocked payment is not a pass against the real PSP.

Decision table

Result	Action
Pass with headroom	Ship; keep watch
Pass at the edge	Ship with tighter alerts / lower campaign load
Fail on latency only	Optimize hot path / indexes
Fail on errors + DB connections	Fix pool budget before marketing traffic

What the test still cannot prove

• Rare race conditions
• Multi-hour memory leaks (need soak)
• Vendor outages
• Data-volume growth next quarter

Schedule soak separately: lower RPS, longer duration, watch RSS and pool wait.

How to write criteria that survive debate

Bad criteria are vague (“should be fast”). Good criteria are checkable in CI or a report:

PASS if:
  - duration >= 15m at target RPS
  - http_p95_ms <= 300
  - error_rate <= 1%   (5xx only; count 429 separately if intentional)
  - db_threads_connected p99 < 0.8 * max_connections
  - pool_wait_p95_ms <= 20
FAIL otherwise; do not ship campaign / raise HPA max

Put that block in the PR description for the load-test job. Arguments after the run become “did we meet the block,” not “I feel like 400 ms is fine.”

Environment parity checklist

Before trusting a pass:

Knob	Match prod?
pool_size / max replicas	required
DB size and indexes	required for query tests
cache warm vs cold	label both runs
feature flags	same defaults as prod
CPU/memory limits	request/limit close to prod
third-party calls	real or contract-tested mock

A pass on a laptop Docker Compose with pool_size=50 proves almost nothing about a pool_size=11 production budget.

Interpreting the lab’s two runs

The demo does not open sockets; it encodes the decision function:

• baseline_pool_ok: p95 and errors inside budget → pass → ship with monitoring
• pool_starved: p95 and errors outside budget → fail → fix pools before load

When you replace the toy sampler with k6 output, keep the same function. The tool is interchangeable; the criteria are not.

Soak and spike are different tests

Test	Question	Duration
Spike	Survive a sudden 3–5× burst?	minutes
Capacity	Hold target RPS within SLO?	10–30 min
Soak	Leak memory/connections?	hours

Do not claim “we load tested” if you only ran a 60-second spike once.

Summary

Step	Output
Question	one sentence risk
Criteria	p95, errors, duration, config
Model	traffic shape + fakes labeled
Parity	pools, data, flags match prod
Result	pass/fail → decision

Load testing is decision support. If nobody’s plan would change based on the number, do not burn the cluster generating it.