Payment Recovery Pipeline and Runbooks (Payment Recovery Pipeline And Runbooks)

The previous part showed how correlation by payment id, the step event log, and deferred finalize metrics feed operations. This part covers the recovery pipeline built on that visibility: when a payment gets stuck, what can the system do on its own — and when does a human step in?

The principle of a good recovery pipeline is simple: automation first. The reconciliation worker, stuck watcher, retry worker — these clear most drift without human intervention. Human runbooks only activate when automation hits uniqueness walls or ambiguous evidence.

Payment stuck
  → automatic: reconciliation scan
  → automatic: retry with backoff
  → automatic: PSP status query
  → human: uniqueness wall / ambiguous evidence

Where the concepts first show up

📦 Recovery pipeline
A sequence of stages that automatically diagnose and attempt to fix stuck or inconsistent payments.

📦 Uniqueness wall
A state where a database uniqueness constraint blocks a safe replay or heal attempt.

📦 Evidence-driven runbook
An operational guide where every step defines which evidence (step log, PSP query, audit) must exist first.

📦 Manual review queue
A visible holding area where records automation couldn't resolve accumulate.

A runbook answers 'what to do'; an evidence-driven runbook answers 'what must never be done without this evidence'.

Automation first: layers of the recovery pipeline

The recovery pipeline isn't one worker — it's complementary layers. Each layer handles what the previous one couldn't; none should try to do the next layer's job.

Layer 1: Retry worker
  → retries transient failures with backoff

Layer 2: Stuck watcher
  → resets records stuck in Processing after lease expiry

Layer 3: Reconciliation sweeper
  → compares aged FinalizePending / Expired records against the PSP

Layer 4: Manual review queue
  → records automation couldn't resolve

Layers 1–3 are fully automatic and sufficient most of the time. Layer 4 isn't pipeline failure — it's the visible boundary of what automation can safely do.

The uniqueness wall: where automation stops

A reconciliation worker sees 'Succeeded' from the PSP and tries to move the local record to Captured — but a Captured row with the same idempotency key already exists. The insert or update fails; automation stops.

Reconciliation: payment #5521 → PSP says Captured
  → local UPDATE attempt
  → UNIQUE constraint violation on idempotency_key
  → automation stops
  → write to manual review queue

This isn't a bug — it's a protection mechanism. The uniqueness wall prevents double finalize — but it also stops automation from declaring 'fixed'. That's exactly when an evidence-driven runbook takes over.

Evidence-driven runbook: procedure, not reflex

When human intervention is needed, the runbook follows this sequence:

1. Read the step event log (by payment id)
2. Run a PSP status query (via the provider gateway)
3. Compare local records (payment, order, idempotency)
4. Fill the evidence table
5. Decide: heal / refund / no-action
6. Write an audit record

Every decision point in the runbook requires evidence. 'PSP says Captured but local says Expired' → heal candidate. 'PSP Not Found, local Processing' → not a refund candidate; wait or escalate. 'Two Captured rows, different idempotency keys' → double charge; refund one.

Manual review queue: visible waiting

Records automation couldn't resolve must not disappear silently. The manual review queue is where they accumulate, age, and get prioritized.

Field	Purpose
paymentId	Correlation
stuckReason	UniquenessWall / AmbiguousEvidence / PSPUnknown
evidenceSummary	Step log + PSP query summary
age	How long it's been waiting
priority	Amount, customer complaint, SLA

Every record in the queue should map to a runbook step; the operator should be able to read 'what to do' from the queue itself.

Runbook example: after a uniqueness wall

State: reconciliation couldn't write Captured — idempotency_key conflict

Gather evidence:
  □ Step log: was FinalizeAttempted twice?
  □ Which webhook produced the existing Captured row?
  □ PSP query: how many charges for this correlation id?

Decision tree:
  → One PSP charge, two local rows → close stale row with audit
  → Two PSP charges → refund one (runbook: duplicate charge)
  → No PSP charge → local Captured is wrong → escalate

Distinctions that get blurred

❌ Manual review queue means automation failed
✓ Manual review queue means automation's boundary is visible and safe

❌ A runbook is senior engineer intuition
✓ A runbook is evidence-based, repeatable procedure

❌ The uniqueness wall should be removed
✓ The uniqueness wall should stay; the runbook manages what's beyond it

Automation vs human intervention

Situation	Automation	Human runbook
Transient timeout	Retry worker	Not needed
Aged FinalizePending	Reconciliation	Not needed
Idempotency conflict	Stops, writes to queue	Gather evidence, decide
Ambiguous PSP response	Stops, writes to queue	Escalate or wait

Recovery pipeline checklist

1. Are retry, stuck watcher, and reconciliation layers separate and sequential?
2. Does a uniqueness constraint violation automatically write to the manual review queue?
3. Does every runbook step explicitly define which evidence it requires?
4. Are records in the manual review queue sorted by age and priority?
5. Is an audit record and step event log entry mandatory after human intervention?
6. Is the automation resolution rate tracked as a metric (automation_resolution_rate)?

What to take away

1. The recovery pipeline is layered with automation first; humans are the last resort.
2. The uniqueness wall stops automation — that's protection, not a bug.
3. Runbooks must be evidence-driven; reflexive action is dangerous.
4. The manual review queue keeps unresolved records from disappearing silently.

A good recovery pipeline doesn't try to eliminate human intervention — it confines human intervention to the right moment, with the right evidence, following the right procedure.

The next part goes into the messaging layer: exactly-once is a lie — and how effectively-once business outcomes are achieved through defense in depth.