Structured logging that shortens on-call instead of filling disks

On-call does not fail because the system produced zero logs. It fails because five minutes of scrolling cannot answer: which tenant, which request, which dependency, how long did we wait? Structured logging is not a fashion choice. It is a contract between the process that emits lines and the human who will grep them at 3 a.m.

This article locks a minimal field set for an HTTP service, shows a bad line next to a useful one, and includes a tiny demo you can run from the repo.

The noisy default

Many services still emit lines like:

[INFO] handling request for user and order stuff status=503 took a while

Problems stacked in one sentence:

• No request id → cannot join with gateway or worker logs.
• No tenant / order id → cannot scope blast radius.
• No latency breakdown → “took a while” is not actionable.
• Free text shape → every dashboard regex is a science project.
• Status buried in prose → hard to alert on http_status >= 500.

When the database pool times out, that line tells you almost nothing you did not already know from the user report.

A minimal field contract

For request-scoped logs on a backend API, keep a stable JSON object:

Field	Why it exists
ts	Order events without relying on ingest order
level	info / warn / error for paging rules
msg	Stable event name (http_request), not a novel
service	Multi-service greps
request_id	Join across hops
route	Low-cardinality path template, not raw URLs with ids
http_status	Outcome
latency_ms	User-visible cost
tenant_id	Blast radius
order_id (or resource id)	Entity under repair
error_code	Machine-stable reason when failing
db_wait_ms (optional)	Proves pool/query delay vs pure CPU

Cardinality rules:

• High cardinality (ids, emails) belongs in fields you filter occasionally—not in msg.
• Route templates stay low cardinality (POST /v1/orders/{id}/pay), so you can chart error rates by route.

Bad vs good (runnable)

node content/labs/structured-logging-on-call/demo.mjs

The demo prints the same two logical events twice: once as noisy text, once as JSON. The failure case carries error_code: "db_pool_timeout" and db_wait_ms: 1180.

Useful on-call query sketch:

level:error AND error_code:db_pool_timeout AND tenant_id:t_42

Or, when you only have a request id from the client:

request_id:req_01JDEF

That second query should pull gateway, API, and worker lines if every hop forwards the same id.

Incident walkthrough with the good shape

Synthetic timeline:

1. Page: elevated 503 on orders-api.
2. Filter service:orders-api AND level:error for the last 15 minutes.
3. Facet on error_code → dominant db_pool_timeout.
4. Facet on tenant_id → mostly t_42 or evenly distributed (local vs global).
5. Check db_wait_ms vs latency_ms → wait dominates, so look at pool/DB, not JSON serialization.
6. Grab one request_id and confirm the same code on dependencies.

None of those steps work cleanly if the only log says “took a while.”

What to stop logging

Structure is not “log everything as JSON.” Volume and privacy still matter.

Stop or sample:

• Successful high-QPS health checks at info (use metrics).
• Full request/response bodies on hot paths.
• The same warning every loop iteration without backoff.

Never log:

• Passwords, session tokens, raw card data, password reset links.
• Unbounded free-text exception strings that may embed PII from upstream.

Prefer metrics for:

• QPS, saturation, pool active count—time series beats log counting for capacity.

Logs answer which entity and which code path. Metrics answer how bad and is it still rising.

Error logging rules that prevent spam

1. One error log per failed request at the edge of the service (plus optional debug context), not one per layer with different formats.
2. Stable error_code values (db_pool_timeout, payment_declined), not interpolated English sentences as the only key.
3. Attach the ids you already have; do not require a second query to the database just to log.
4. Downgrade expected client errors (validation 400) to info/warn so pages stay reserved for 5xx and SLO burn.

Implementation sketch

Pseudocode for a request logger:

function logRequest(ctx, result) {
  logger.write({
    ts: new Date().toISOString(),
    level: result.httpStatus >= 500 ? "error" : "info",
    msg: "http_request",
    service: "orders-api",
    request_id: ctx.requestId,
    route: ctx.routeTemplate,
    http_status: result.httpStatus,
    latency_ms: result.latencyMs,
    tenant_id: ctx.tenantId,
    order_id: ctx.orderId,
    error_code: result.errorCode,
    db_wait_ms: result.dbWaitMs,
  });
}

Enforce the schema in code review or with a lightweight test that parses a sample line and asserts required keys exist on error paths.

Summary

Practice	Purpose
Stable JSON fields	Query without poetry
request_id everywhere	Join hops
Low-cardinality route	Chart and alert
error_code + timing fields	Jump to the subsystem
Drop bodies / secrets / health spam	Keep signal affordable

Structured logging shortens on-call when every critical failure line can answer who, what resource, which request, which dependency, how long. The demo in this repository is small on purpose: if your production lines cannot support the same query, fix the contract before buying another dashboard.