Optimistic Concurrency Under Webhooks (Optimistic Concurrency Under Webhooks)

The previous part showed why eventual consistency is unavoidable: the PSP can never join your transaction protocol, and saga plus reconciliation is the real answer. This part focuses on the busiest moment inside that inconsistency window — when a webhook and a synchronous response touch the same payment record at the same time.

The checkout orchestrator sends a charge request; the provider gateway gets a response from the PSP. At the same moment — sometimes milliseconds before, sometimes after — a webhook arrives for the same payment. Both paths can carry correct information; if both try to write at once, the result is either a lost update or, worse, a stale read that sends the client a secret or redirect URL for a payment that has already reached a terminal state.

Sync response ──► Payment #42 (version=3) ──► Captured
Webhook       ──► Payment #42 (version=3) ──► Captured (again?)
                      │
                      ▼
              Version token + lease
              → one winner writes

Optimistic concurrency here isn't a performance optimization — it's the mechanism that keeps wrong data from reaching the client on a terminal payment.

Where the concepts first show up

📦 Version token (optimistic lock)
A counter that increments on every update; a write succeeds only when the expected version matches.

📦 Lease
A time-bounded claim that gives one worker the right to process a specific payment record.

📦 Stale read
A version read during processing that is no longer valid by the time the write happens.

📦 Terminal payment
A final state with no return path: Captured, Failed, or Refunded.

A lease says 'I'm processing this record'; a version token says 'I'm still seeing this version'. Together they stop the webhook path and the synchronous path from overwriting each other.

Two paths, one record: where the race starts

In a redirect-based payment, the synchronous path usually returns Pending; the real outcome arrives via webhook. In a card payment, both paths can carry Captured or Failed — and both can arrive almost simultaneously. The checkout orchestrator tries to write both into the same payment row.

The classic failure: both handlers read the record, update the status, and save. Last write wins; the update in between silently disappears. The more dangerous scenario: the orchestrator reads an old version before reaching a terminal state and returns a client secret or redirect URL that still looks valid — even though the payment has already completed or failed.

T=0  Orchestrator: sends charge
T=1  Webhook arrives → writes Captured (version 2→3)
T=2  Sync response arrives → had read Pending (version 1)
     → returns redirectUrl to client (stale!)
T=3  Customer follows redirect → payment already Captured

Version token: write only when the version matches

Every payment record carries a monotonically increasing version field. Updates use: UPDATE ... WHERE id = ? AND version = ?. No match means zero rows affected — a signal that another path got there first.

Webhook handler
  READ payment (version=2, status=Processing)
  → status=Captured, version=3
  UPDATE WHERE version=2 ✓ (1 row)

Sync handler (stale read)
  READ payment (version=2, status=Processing)  ← webhook not committed yet
  → status=Captured, version=3
  UPDATE WHERE version=2 ✗ (0 rows — webhook already wrote)
  → re-read, see terminal state, do not return client secret

A version token alone isn't enough; you also need a defined response to a detected stale read: re-read, check for terminal state, return only the current outcome to the client.

Lease: claim webhook processing rights for a bounded time

Before touching the record, the webhook handler acquires a short lease: 'I'm processing Payment #42 for 30 seconds.' While the lease holds, no other worker can process that record in the webhook or recovery flow.

Webhook arrives
  → acquire lease (paymentId, ttl=30s)
  → lease unavailable → defer / retry
  → lease acquired → update with version token
  → release lease

A lease stops the same webhook from being processed by two workers at once. A version token resolves collisions between different paths (sync vs webhook). They answer different problems and must be used together.

Client secret leakage on terminal payments

The most serious stale-read scenario is returning a client secret or redirect URL after the payment has reached a terminal state. Returning Pending + redirectUrl after the payment is Captured leads to an unnecessary second charge attempt or customer confusion.

The rule is simple: never return a client secret, redirect URL, or retry token for a payment in a terminal state. When a handler gets a version conflict or suspects a stale read, it re-reads the record; if it sees a terminal state, it returns only the final outcome.

State	Returned to client
Processing, redirect needed	redirectUrl (valid)
Captured (terminal)	Success outcome, no secret
Failed (terminal)	Failure outcome, no secret
Version conflict → re-read → Captured	Success outcome, no secret

Distinctions that get blurred

❌ Pessimistic locking is always safer
✓ A short lease plus version token resolves the race while preserving throughput

❌ A version conflict means throw an exception
✓ A version conflict means another path won — re-read and align with current state

❌ Lease and version token do the same thing
✓ Lease blocks parallel processing of the same record; version token blocks lost updates

Lease vs version token

Criterion	Lease	Version Token
Prevents	Parallel processing of same record	Lost update
Duration	Bounded by TTL	Persistent, increments on every write
On conflict	Wait / defer	Re-read / retry

Optimistic concurrency checklist

1. Are payment updates performed with a WHERE version = ? condition?
2. On version conflict, does the handler re-read and check for terminal state?
3. Is returning a client secret or redirect URL on terminal state blocked at code level?
4. Does the webhook handler acquire a lease before processing?
5. Is the lease TTL longer than the P99 webhook processing time?
6. Do the sync response handler and webhook handler share the same finalize logic?

What to take away

1. Webhook and sync paths touch the same record concurrently; optimistic concurrency is the standard answer to that race.
2. A version token prevents lost updates; a lease prevents parallel processing of the same record.
3. A version conflict isn't an error — it's a signal to re-read.
4. Returning a client secret from a stale read on a terminal payment is a quiet security and UX failure.

You don't need pessimistic locking to resolve the race — you need a disciplined version token plus lease combination that keeps stale reads from reaching the client.

The next part moves to observability so you can actually see these races and finalize steps: correlation by payment id, a step-by-step event log, and metrics for deferred finalize.