Designing a Production Payment Engine (Designing A Production Payment Engine)

This series ran from provider abstraction through webhook reliability, idempotency, sagas, leases, reconciliation, observability, recovery pipelines, and effectively-once processing. This final technical part is the synthesis: if you're designing a payment engine for production, which decisions do you make, and in what order?

The checklist isn't a feature list. Each item is a measurable test of a principle defended across one or more parts of the series. Not 'do we have idempotency' — but 'do the idempotency key, webhook dedup, and DB uniqueness constraint work together'.

Production Payment Engine
  ├─ Boundaries (orchestrator ↔ gateway)
  ├─ State & Evidence
  ├─ Reliability (retry, lease, outbox)
  ├─ Recovery (reconciliation, runbooks)
  └─ Observability (payment id, step log, metrics)

Where the concepts first show up

📦 Checkout orchestrator
The service that manages payment intent, sees semantic results, and knows no PSP detail.

📦 Provider gateway
The service that owns the PSP SDK, webhook translation, and provider-specific flows.

📦 Production readiness
The system behaves correctly not just on the happy path, but during faults, races, and drift.

📦 Architectural checklist
Design decisions turned into measurable yes/no tests.

A production payment engine isn't 'the charge API works' — it's answering 'what happens when the charge fails, the webhook arrives late, or the worker crashes'.

1. Boundaries: no SDK leakage

• Does the checkout orchestrator import zero PSP SDK types?
• Does the orchestrator only see semantic ChargeRequest / ChargeResult?
• Do webhooks reach downstream as semantic events, not raw provider payloads?
• Is the provider gateway the sole owner of the provider → semantic event mapping table?
• Does adding a new PSP require zero lines changed in the orchestrator?

This section is the essence of parts 9–10. If the boundary breaks, every other layer is built on that leak.

2. State and evidence: state ≠ evidence

• Is the payment state machine explicitly defined (Processing, FinalizePending, Captured, Failed, Expired)?
• Are terminal states irreversible?
• Is the payment snapshot (amount, currency, cart) immutable?
• Is evidence from the PSP (webhook, sync response) stored in a separate evidence table?
• Do state transitions rest on evidence, not assumption?

3. Reliability: retry, lease, outbox

• Is a failure taxonomy defined (BusinessDecline, Timeout, RateLimited, Infrastructure)?
• Does each category get its own retry policy?
• Are DB-backed jobs processed with leases?
• Does the webhook handler use lease + version token together?
• Are state changes and events published in the same transaction via the outbox pattern?
• Do the idempotency key, consumer dedup, and DB uniqueness constraint work together?

4. Recovery: automation first, runbook second

• Does the reconciliation sweeper scan aged FinalizePending / Expired records?
• Can orphan charges be resolved via correlation id?
• Is there cart locking for multi-intent scenarios?
• Does the uniqueness wall write to the manual review queue?
• Is the runbook evidence-driven (step log + PSP query + audit)?
• Are heal/refund decisions procedural, not reflexive?

5. Observability: payment id as spine

• Does every log, trace, and metric carry payment id?
• Is the step event log append-only and covering every meaningful step?
• Are deferred_finalize_count and deferred_finalize_age_seconds defined?
• Does drift count alert on sudden spikes?
• Can you trace the full path from checkout to terminal state with one payment id?

6. Consistency model: eventual, measured, observable

• Was saga + reconciliation consciously chosen over 2PC?
• Does every saga step have a defined compensating action?
• Is the eventual consistency window measured and shared with the product team?
• Does the system target 'consistent within a short bounded time' rather than 'consistent at every instant'?

When the checklist is done, ask one final question:
  'What happens on this system's worst day?'
  → The answer should be written in runbooks, metrics, and reconciliation.

Distinctions that get blurred

❌ Checklist = features are done
✓ Checklist = measurable test of an architectural principle

❌ Production ready = load test passed
✓ Production ready = correct behavior during faults is proven

❌ More PSPs = more complexity everywhere
✓ With good boundaries, a new PSP only affects the gateway

Checklist sections and series mapping

Section	Series parts	Core question
Boundaries	9–10	Does the orchestrator know the PSP?
State & evidence	1–4, 6	Does state rest on evidence?
Reliability	5, 7–8, 11, 17, 20	What happens during a fault?
Recovery	12–16, 19	How is drift caught?
Observability	18	Is a stuck payment visible?
Consistency	3, 16	2PC or saga?

Production readiness assessment

1. Walk through all six checklist sections in a design review; answer each item yes/no/partial.
2. Add 'partial' answers to a technical debt list; prioritize by business impact.
3. Run a tabletop exercise for the worst day (PSP 5xx, delayed webhook, worker crash, orphan charge).
4. Note which checklist items were empty during the exercise — those are your first improvement targets.
5. Keep the checklist as a living document; update the relevant item after every production incident.

What to take away from this series

1. A production payment engine is measured by fault-time behavior, not the happy path.
2. The checklist is the measurable synthesis of 22 parts — not a feature list.
3. If boundaries (orchestrator ↔ gateway) break, everything else is built on that leak.
4. The answer to 'what happens on the worst day' should be written in runbooks, metrics, and reconciliation.

Designing a production payment engine isn't writing a charge API — it's making the gap between capture and complete controlled, observable, and recoverable.

The final part shifts to career perspective: what fintech companies actually hire for.