Debugging a production timeout when there is no stack trace

Timeouts often arrive without a stack trace. The client sees 504, the load balancer says upstream timed out, and your process is still healthy enough to answer /healthz. Local CPU flame graphs look fine. The useful question is not “why is Node broken?” but which hop spent the budget.

This article recreates that failure in a small lab: an overall request budget, three dependency calls, and structured logs that name the failing step without any exception stack.

Symptom

Synthetic checkout request:

• Client: HTTP 504 / gateway timeout
• App log: request_end with error_code=payment_timeout
• Liveness: still green
• No uncaught exception, no core dump

If you only have “latency high” on a single RED chart, every dependency is a suspect. You need per-hop timing.

Lab recreation

node content/labs/debug-timeout-without-stack-trace/demo.mjs

The demo runs three steps under budgets:

Step	Work latency	Budget	Result
auth	~12 ms	100 ms	ok
inventory	~35 ms	100 ms	ok
payment	~900 ms	200 ms	payment_timeout

Overall budget is 500 ms; payment alone exceeds its own 200 ms cap and dominates the failure.

Sample log shape:

{
  "msg": "request_end",
  "request_id": "req_lab_timeout_1",
  "http_status": 504,
  "latency_ms": 250,
  "error_code": "payment_timeout",
  "steps": [
    { "name": "auth", "ok": true, "ms": 12 },
    { "name": "inventory", "ok": true, "ms": 35 },
    { "name": "payment", "ok": false, "ms": 200, "code": "payment_timeout" }
  ]
}

What looked like the cause but was not

Hypothesis	Why it tempts	How the lab kills it
Event loop blocked	Timeouts + Node folklore	auth/inventory finish quickly in same request
Database pool on this service	Recent pool incidents	no DB step; payment is the slow hop
“Need more replicas”	Autoscale reflex	each replica would still wait 900 ms on payment

Without step fields, people scale the wrong tier for a day.

Narrowing technique (no stack required)

1. Grab request_id from the client or edge log.
2. List spans or step logs for that id.
3. Find the first step where ok=false or where ms approaches budget.
4. Open that dependency’s dashboards (payment provider latency, error rate, pool).
5. Only then take stack samples if the slow hop is your code under CPU.

Timeouts are a budget accounting problem first, a profiler problem second.

Instrumentation you want in production

• Deadline / timeout per dependency, not only one global client timeout.
• Log error codes like payment_timeout, inventory_timeout.
• Propagate request_id / trace id on outbound calls.
• Metrics: dependency_latency_ms{hop=payment} and timeout count by hop.

A global 30s HTTP client timeout with no hop labels produces mystery 504s forever.

Fix and follow-ups (for this lab shape)

Mitigations:

• Raise or tune payment budget only if SLO allows; better: fix payment latency.
• Fail fast and show a controlled error instead of waiting for the gateway.
• Circuit-break payment when timeout ratio spikes.

Detection upgrades:

• Alert on error_code=payment_timeout rate, not only aggregate 5xx.
• SLO burn on pay route with dependency breakdown panel.

Regression guard:

• Integration test: stub payment delay > budget ⇒ assert payment_timeout and no double side effects.

Summary

Without stacks you still have	Use it to
request_id	join edge + app + dependency
per-step timing	see which hop ate the budget
stable error_code	page the right owners
dependency metrics	confirm outside your process

The lab’s punchline is deliberate: payment is slow, auth is fine, the process is not wedged. When production looks the same, believe the step logs before you believe a random guess.