Idempotent callbacks for payments and webhooks without a magic framework

Payment providers and internal webhooks retry. They retry because your endpoint timed out, because their network blipped, or because their policy says “at least once.” If your handler charges, ships, or emails on every delivery, duplicates become customer-facing bugs. Idempotency is how you make at-least-once delivery safe.

This article uses a single operation—record a successful charge for an order—and a tiny lab that proves a second delivery does not double-charge.

The business rule

For a given provider event_id, run the side effect at most once. Retries must return the original success payload (or an equivalent success) without repeating money movement.

Key choice: use the provider’s event id (or a documented idempotency key), not only your order_id. One order may legitimately receive multiple different events (auth, capture, refund). Collapsing everything on order_id alone is wrong.

Failure modes you must assume

Delivery pattern	What naive code does
Duplicate POST same event	Double charge / double email
Overlapping concurrent retries	Two workers both pass “not seen” check
Success after provider timeout	Provider retries; you must no-op
Partial failure after side effect	Need transaction or compensation design

Data model

Minimal durable record:

idempotency_keys
- key            (provider event id) PRIMARY
- order_id
- result_json
- created_at

Rules:

1. Insert-first or unique constraint so two workers cannot both create the key.
2. Store the result you will replay on duplicates.
3. Keep records long enough to cover the provider’s retry window (days to weeks—read their docs).

Pseudo-SQL for the safe path:

BEGIN;
-- lock order row if you also update order state
INSERT INTO idempotency_keys(key, order_id, result_json)
VALUES ($event_id, $order_id, NULL)
ON CONFLICT (key) DO NOTHING;

-- if insert affected 0 rows: select result_json and return it

-- perform side effect (charge ledger line, mark paid)
UPDATE idempotency_keys SET result_json = $result WHERE key = $event_id;
COMMIT;

Exact isolation tactics vary by database, but the invariant is: unique key + single winner for the side effect.

Handler flow

1. Verify signature / auth
2. Parse event id + type
3. Begin transaction
4. Try claim idempotency key
5. If already claimed → return stored result
6. Else apply side effect + store result
7. Commit
8. Return 200 with stable body

Return 2xx for duplicates when the original succeeded so the provider stops retrying. Returning 500 on duplicates invites infinite retries.

Lab: double delivery

node content/labs/idempotent-callbacks/demo.mjs

The demo keeps an in-memory map (not for production) and processes the same eventId twice:

first  { chargedCents: 2599, duplicate: false, ... }
second { chargedCents: 2599, duplicate: true, ... }
side effects recorded: 1

Invariant: one stored side effect for one event id. Production replaces the Map with a table and transaction as above.

Races

Two concurrent requests with the same event id:

1. Both pass “select missing” if you only check-then-act without a unique constraint.
2. Fix with unique index on the key and handle conflict by reading the winner’s result.
3. Optionally use SELECT … FOR UPDATE on the order row to serialize state transitions.

Do not rely on “it is unlikely.” Providers retry aggressively under incidents.

What is still not idempotent

Be honest in APIs and runbooks:

• Different event ids that mean the same business action (provider bugs) need business reconciliation, not only a key table.
• Non-transactional side effects (send email, call another HTTP API) need their own idempotency or outbox pattern.
• In-memory labs do not survive multi-instance deploys—durable storage is mandatory.

Tests and runbook

Automated tests:

1. Same event twice → one ledger row / one charge record.
2. Concurrent double POST → still one side effect (stress with parallel clients).
3. Unknown signature → 401 and no claim written.

Runbook:

1. Provider dashboard shows multiple deliveries for one event.
2. Query idempotency_keys by event id.
3. If result stored, customer is safe; explain retries.
4. If missing and money moved, escalate as data incident—process gap.

Summary

Piece	Choice
Key	Provider event id (documented)
Storage	Durable unique row + result
Concurrency	Unique constraint / transaction
Response	2xx replay on duplicates
Proof	Double-delivery test in CI

Idempotency is not a framework feature you toggle. It is a data constraint around a side effect. Start with one callback type, one key, one table, and a test that fails if a second call charges twice—then expand to the rest of your webhook surface.