A service can return HTTP 200 on /healthz and still be useless to every real client. That is not a philosophical problem. It is what happens when the probe only answers “is the process listening?” while business traffic depends on a database, queue, or config that is already down.
This article rebuilds that failure in a small lab, shows why load balancers keep sending traffic, and separates liveness from readiness with code you can run from this repository.
The false green signal
Start with the simplest “healthy” endpoint many services ship on day one:
if (url.pathname === "/healthz") {
send(res, 200, { status: "ok" });
return;
}While that process is up, every probe succeeds. That is exactly what a liveness signal should do: “restart me only if I am wedged.” It is a terrible readiness signal: “send me user traffic.”
In the lab server (content/labs/health-checks-that-lie/server.mjs), the database dependency is intentionally unreachable by default. The business path /orders then returns 503, but /healthz still returns 200.
A representative probe run against the lab looks like this:
| Path | Status | Meaning |
|---|---|---|
/healthz | 200 | Process is alive |
/readyz | 503 | Not ready for traffic |
/orders | 503 | Real users would fail |
If your platform only watches /healthz, the row that matters—/orders—never enters the routing decision.
What the check actually tested
False-green checks usually test one of these and stop:
- The HTTP server accepts connections. The event loop is not completely dead.
- A static string is returned. No dependency I/O at all.
- A shallow self-check. For example “can I allocate a small object?” that never touches production dependencies.
None of those answer the question the load balancer is implicitly asking: If I route this request to you, can you complete the work this service exists to do?
Kubernetes (and most managed load balancers) will keep a target in rotation while the configured probe returns success. With only a liveness-style endpoint wired into the service or target group health check, pods or instances stay Ready / healthy even when the database pool cannot check out a connection. Clients then see elevated 5xx or timeouts, while dashboards still show “all instances healthy.”
That mismatch is the lying health check.
Liveness vs readiness (and why both exist)
Use two different questions:
| Probe | Question | Failure action | Should check DB? |
|---|---|---|---|
| Liveness | Is the process stuck beyond recovery? | Restart the container/process | Usually no |
| Readiness | Should this instance receive traffic right now? | Remove from Service / target group | Yes, for hard dependencies |
| Startup (optional) | Has initial warm-up finished? | Delay other probes | Sometimes |
Putting a database ping on liveness is a common mistake. A brief DB blip then restarts every pod at once, which often makes the outage worse. Putting no dependency check on readiness is the opposite mistake: traffic keeps arriving while the only useful answer is “not now.”
Rule of thumb used in this lab:
/healthz: cheap process liveness. No remote I/O./readyz: fail closed when a hard dependency required to serve the default request path is unavailable.- Business handlers: still handle dependency errors explicitly; readiness is not a substitute for correct application error handling.
Dependencies that must block readiness
Not every dependency belongs on the readiness path.
Block readiness when:
- The service cannot perform its primary operation without it (primary database, required auth service, mandatory schema version).
- Partial availability is worse than temporary removal from the pool (for example, accepting writes that cannot be committed).
Do not block readiness when:
- The dependency is optional (analytics sink, non-critical cache).
- Failure should degrade features but still allow a useful subset of traffic.
- Checking it is so slow or flaky that probes themselves cause overload.
For this lab, the database is a hard dependency of GET /orders, so it belongs on /readyz.
Implementation
The readiness handler in the lab performs an explicit dependency check and returns 503 until the check passes:
if (url.pathname === "/readyz") {
const db = await checkDatabase();
if (!db.ok) {
send(res, 503, {
status: "not_ready",
checks: { database: db },
});
return;
}
send(res, 200, {
status: "ready",
checks: { database: db },
});
return;
}A few implementation details that matter in production systems as well as the lab:
- Timeout every dependency probe. A hung readiness check becomes a hung kubelet probe. Bound it (tens to a few hundred milliseconds for simple pings).
- Return structured reasons in the body for humans, but keep the status code as the contract platforms understand (200 vs 503).
- Avoid cascading probe storms. If you have 100 replicas, a heavy readiness query every second is a self-inflicted load test. Prefer cheap checks (
SELECT 1, connect+ping) over full business queries. - Cache readiness failures briefly only if you understand the trade-off. A one-second negative cache can protect the database; a thirty-second cache can hide recovery.
Example Kubernetes shape (illustrative):
livenessProbe:
httpGet:
path: /healthz
port: 3000
initialDelaySeconds: 10
periodSeconds: 10
readinessProbe:
httpGet:
path: /readyz
port: 3000
initialDelaySeconds: 5
periodSeconds: 5
failureThreshold: 2Wire the Service / load balancer to respect readiness (Kubernetes does this for Ready pods automatically). If you only configure a cloud target-group health check, point that check at /readyz, not /healthz.
Reproduce the lab
From the repository root:
# Terminal A — dependency down (default)
node content/labs/health-checks-that-lie/server.mjs
# Terminal B
node content/labs/health-checks-that-lie/probe.mjsYou should see /healthz succeed while /readyz and /orders fail. That is the false-green pattern in one screen of output.
Simulate recovery without rewriting the process:
# Terminal A
DATABASE_OK=1 PORT=3001 node content/labs/health-checks-that-lie/server.mjs
# Terminal B
BASE_URL=http://127.0.0.1:3001 node content/labs/health-checks-that-lie/probe.mjsWith DATABASE_OK=1, /readyz and /orders should return 200 together. Force the dependency down again with FAIL_DB=1 even if DATABASE_OK=1. The important contract is that readiness and the business path agree.
The test that catches regressions
Do not rely on “we remembered to configure the probe.” Add an automated assertion in CI or a smoke suite:
// Pseudocode for a staging smoke test
assert.equal((await fetch(base + "/healthz")).status, 200);
// With DB blocked (fault injection, toxiproxy, or FAIL_DB in the lab):
assert.equal((await fetch(base + "/readyz")).status, 503);
assert.equal((await fetch(base + "/orders")).status, 503);
// Platform-level check (where you can):
// Ready pod count drops / target leaves healthy set while FAIL_DB=1The regression you are guarding against is silent: someone “simplifies” /readyz back into return 200, production still deploys, and the next database incident looks like a mysterious wave of application 5xx with green health panels.
Trade-offs and edge cases
Readiness flapping. If the database is intermittently slow, aggressive thresholds bounce instances in and out of the pool. Prefer slightly longer windows (failureThreshold, slower periodSeconds) over removing the check entirely.
Multi-dependency services. Aggregate hard checks: any hard failure → 503. Report each child status in the JSON body so on-call can see which dependency failed without tailing logs first.
Admin-only processes and workers. A queue worker may not need an HTTP readiness probe for user traffic, but it still needs a clear “should this replica consume jobs?” signal—often the same dependency gates applied before taking work from the queue.
Security. Readiness endpoints should not require end-user auth (the platform must call them), but they also should not dump secrets or internal hostnames into public responses.
Summary
/healthzmeans “don’t kill me yet.” Keep it cheap./readyzmeans “you may send me traffic.” Fail closed on hard dependencies.- Platforms route on the probe you configure. If that probe is a lying 200, users discover the outage first.
- Prove the contract with a test that breaks the dependency and expects readiness and business paths to fail together.
The lab in this repository is intentionally small. The production version of the same design is mostly discipline: name the hard dependencies, put them on readiness, keep liveness boring, and never let “status: ok” mean more than it measured.