Webhook Reliability in Payment Systems (Webhook Reliability In Payment Systems)

A webhook is not a guaranteed message

A webhook the PSP sends you carries no guarantee that “this event will arrive exactly once, in the right order.” Quite the opposite — it can break in four different ways: the same event can arrive more than once, an event can never arrive at all, events can arrive in a different order than they were sent, and an event can show up minutes late.

PSP → Webhook
  ⚠ Duplicate: the same event twice
  ⚠ Missing: the event never arrives
  ⚠ Out-of-order: a capture notification arrives before the authorize notification
  ⚠ Delayed: an event arrives minutes late

This part covers how to make your webhook handler resilient against all four of these scenarios.

Concepts, defined where they first appear

📦 Webhook
An HTTP call an external system (the PSP) makes to your endpoint to report an event that happened on its side.

📦 Signature Verification
A cryptographic check confirming an incoming webhook really came from the PSP and that its content wasn't altered.

📦 At-least-once Delivery
A delivery model that guarantees an event arrives at least once, sometimes more — but makes no guarantee about order or duplicate count.

📦 ACK (Acknowledgement)
The fast HTTP response (usually 200) a webhook receiver sends to tell the PSP the event was received.

📦 Durable Write
Writing the event to persistent storage no matter what happens next — not just holding it in memory.

All of these concepts serve one rule: your webhook handler must safely record the incoming event first, no matter what, and only do the heavy work afterward.

Duplicate: the same event arrives twice

If the PSP doesn't get a timely 200 back from you (network issue, server slowness), it resends the same event. This isn't a bug — it's the natural consequence of the PSP's at-least-once guarantee. The event ID + inbox pattern we detailed in part five is your first line of defense here.

Webhook #1: evt_001 → processed
Webhook #2: evt_001 (redelivery) → already in the inbox, processing skipped, 200 returned

Missing: the event never arrives

Sometimes a webhook never shows up: a network outage, an error on the PSP's side, or your endpoint being temporarily unreachable. A system that relies solely on webhooks stays stuck at “I don't know” forever in that case. That's why the webhook should be your primary notification channel, not your only source of truth; periodic reconciliation against the PSP's status query API should always stand as a secondary safety net.

Webhook (primary, fast)
  + Periodic status polling (secondary, slow but guaranteed)
  = even if the webhook is lost, the truth gets caught sooner or later

Out-of-order: events arrive out of sequence

The network layer makes no guarantee that events arrive in the order they were sent. A “captured” notification, for example, can arrive before the “authorized” notification that was supposed to precede it. If your handler updates the state machine without checking the event's timestamp or version, this is exactly where the guards from part two need to kick in.

Arrives: payment.captured (t=2)
Arrives: payment.authorized (t=1, but got here later)
  → guard: a t=1 event can't roll back a state that already reached t=2; it's silently rejected

Delayed: the event arrives late

A webhook can show up minutes late due to queuing on the PSP's side or processing delay on yours. In this case, your handler needs to distinguish “now” from “when the event actually happened”; if business decisions are made based on when the event was processed rather than its own timestamp, ordering bugs compound.

Why you should never run heavy work synchronously

Your webhook handler should do nothing beyond signature verification, minimal validation, and a durable write (into the inbox). Steps like decrementing stock, opening a ledger entry, or sending a notification should always be handed off to a separate background job.

Webhook Handler (fast, synchronous)
  1. Verify the signature
  2. Do a minimal schema check
  3. Write the event to the inbox (durable)
  4. Return 200 OK

Background Job (slow, asynchronous)
  5. Read the event from the inbox
  6. Apply the actual business decisions (stock, ledger, notification)

This split matters for two reasons. First, PSPs usually enforce a short timeout on the webhook response (a few seconds); if heavy work exceeds that window, the PSP treats the request as failed and resends it, which only inflates your duplicate count. Second, running heavy work synchronously makes your webhook handler dependent on the performance of downstream systems (a slow stock service, say) — and that dependency can cause the webhook call itself to time out.

The mappings that get confused most often

❌ A webhook is the single, reliable source of truth
✓ A webhook is the primary notification channel; periodic status polling is the secondary safety net

❌ Returning 200 means the work is done
✓ 200 means the event was safely received; finishing the work is a separate, asynchronous step

❌ Webhooks always arrive in the order they were sent
✓ Order is never guaranteed; without guards, the state machine can drift backwards

❌ Signature verification is optional if you have an IP allowlist
✓ Signature verification is the real defense against forged or tampered webhooks

A checklist for auditing your webhook handler

1. Does your webhook handler verify the signature on every request, or does it just trust an IP allowlist?
2. What heavy work does your handler run synchronously before writing the event to the inbox?
3. If a webhook never arrives, how many hours or days does it take your system to notice? Do you have a reconciliation job?
4. Can two out-of-order webhooks push your state machine into an invalid state? Have you tested your guards?
5. What's your PSP's webhook timeout window, and how much of that window does your handler actually use?

If you answer “no, we haven't checked” to even one of these five questions, your webhook reliability is probably resting on an untested assumption.

What to take away from this part

1. Webhooks can be duplicate, missing, out-of-order, and delayed; your handler should assume all four as the default, not the exception.
2. Signature verification is the real line of defense against forged webhooks; an IP allowlist alone isn't enough.
3. The handler should ACK fast and hand heavy work off to an asynchronous job; synchronous heavy work inflates both timeout and duplicate risk.
4. A webhook is the primary channel, not the sole source of truth; periodic status polling should always stand as a secondary safety net.

Trusting that a webhook “will arrive” isn't a design. Designing for “it might not arrive, it might arrive twice, it might arrive out of order” is.