Idempotency Beyond API (Application Programming Interface) Requests: A Layered Defense (Idempotency Beyond API Requests)

Idempotency is a stack, not a header

Most teams learn idempotency as “add an Idempotency-Key header to the API request” and stop there. In payment systems, that's just the visible tip of the iceberg. The same operation can pass through your system as a “retry” from at least five different points: the client's retry, the PSP's webhook redelivery, the message broker's at-least-once delivery, the worker picking up the job again, the saga step re-running.

Client retry → API idempotency key
PSP retry   → Gateway event id
Broker retry → Inbox record
Worker retry → Job uniqueness
Saga retry  → Step marker

This part covers how to build idempotency separately at each of these five layers, not just at one point.

Concepts, defined where they first appear

📦 API Idempotency Key
A unique key sent by the client that guarantees the same result if the same request is sent again.

📦 Gateway Event ID
A unique identifier the PSP attaches to every webhook or notification; used to distinguish multiple deliveries of the same event.

📦 Inbox
A durable table where an incoming event is recorded before processing, filtering out duplicates.

📦 Job Uniqueness
A constraint that prevents a background job from being queued a second time under the same job key.

📦 Step Marker
A durable record marking a saga step as complete, preventing it from re-running.

These five concepts don't substitute for each other. An API idempotency key prevents duplication between the client and your API — it has no effect on the PSP's webhook, your broker's delivery, or your worker's job.

Layer 1: the API request

If a customer clicks “Pay” twice (network lag, a double click), the client sends the request with the same Idempotency-Key. If the server has already seen that key, it doesn't re-run the operation — it returns the result of the first run. This layer is purely a contract between your API and the client.

POST /payments  Idempotency-Key: abc123
  → first call: operation runs, result is stored
  → second call, same key: stored result is returned, operation does not re-run

Layer 2: the event from the PSP

The PSP can send the same event (say, “capture succeeded”) more than once, due to network issues or its own retry policy. Each of these events carries a unique event ID from the PSP. If you've already seen that ID, you shouldn't reprocess the event — but you still need to explicitly acknowledge it back to the PSP (ACK).

Webhook #1: event_id=evt_001, type=payment.captured
Webhook #2: event_id=evt_001, type=payment.captured  (redelivery)
  → if the event_id has been seen, skip processing, return 200 OK

Layer 3: the message queue / inbox

If the event ID check happens directly inside the code that processes the event, you risk a race condition where two different workers process the same event at once. That's why the inbox pattern exists: the event is first written to an inbox table under a unique constraint; if that write fails (it already exists), the event has already been seen and processing is safely skipped.

INSERT INTO inbox (event_id, ...) VALUES ('evt_001', ...)
  → success: seen for the first time, queue the work
  → unique constraint violation: already seen, silently skip

Layer 4: the job itself

After the inbox, the work is handed off to a background job. That job can also end up queued more than once on its own (a retry mechanism, a redeploy). The job queue should reject a second job that shares the same job key (for example payment_id + step_name).

Job key: payment_id=pay_42, step=finalize_stock
  → a second job attempt with the same key: rejected at the queue level

Layer 5: the saga step itself

Even while the job runs, the step itself must be idempotent — because the job can restart after a crash. This final layer is the step marker we introduced in part three: every step durably marks its own completion, and if the step is called again, it checks that marker instead of redoing the real work (decrementing stock, opening a ledger entry).

Step marker: finalize_stock=DONE (payment_id=pay_42)
  → if the step is called again, the marker is checked, the work isn't redone

Why all five layers are needed separately

Each layer answers a different question: who's retrying — the client, the PSP, the queue, the worker, or the step itself. If you build idempotency only at the API layer and skip the other four, you can end up with a finalization saga that runs “once at the API, but three times in the background” — and you usually only discover this in production, often via a customer complaint.

The mappings that get confused most often

❌ Adding an Idempotency-Key header solves the idempotency problem
✓ That header only solves duplication at the client-API layer

❌ Checking the PSP's event ID alone is enough
✓ Event ID checking needs an inbox / unique constraint to guard against race conditions

❌ If the job queue does at-least-once delivery, the job automatically becomes idempotent
✓ At-least-once delivery plus a non-idempotent job equals unsafe; the job itself must be idempotent

❌ Calling a saga step again after it already succeeded is harmless
✓ Without a step marker, calling it again triggers the side effect (stock decrement, payment) a second time

A checklist for auditing your idempotency stack

1. Does your API have an idempotency key? If so, how long do you keep the stored result?
2. Do you check the event ID on PSP webhooks, or do you process every webhook directly?
3. Does your inbox table have a unique constraint on event ID, or is the check done in application code (a race-condition risk)?
4. Does your job queue reject a second job with the same job key?
5. Does every saga step check a marker for its own completion, or does it redo the work every time it runs?

If two of these five layers are missing, your system is probably exposed to rare-but-recurring double-processing bugs.

What to take away from this part

1. Idempotency is not one header or one check; it's a stack that needs to be built separately across at least five independent layers.
2. Each layer answers a different source of retries (client, PSP, queue, worker, saga step); none of them substitutes for another.
3. The inbox pattern is a structural solution that makes event ID checking safe against race conditions — it's not just an “if” check.
4. Under an at-least-once delivery model, a non-idempotent work step will eventually run twice.

Reducing idempotency to a single header is like putting one door on a five-story building and leaving every other floor's window wide open.