Retry Algorithms for Payment Workers (Retry Algorithms For Payment Workers)

The previous part defined four failure categories. This part builds the actual algorithm for the three that are retry-eligible (timeout after a status check, rate limited, infrastructure): how long do we wait, how many times do we try, and when do we stop entirely and open a circuit breaker?

Attempt 1 → fails → wait (backoff) → Attempt 2
  → fails → wait (longer) → Attempt 3
  → fails → cap reached → defer / dead-letter

Two verbs shouldn't get conflated here: retry means trying again inside the same worker almost immediately; defer means putting the job back and picking it up much later. Both look like 'try again', but the timing and the responsibility differ.

Where the concepts first show up

📦 Exponential backoff
A strategy that doubles the wait time on every attempt: base * 2^attempt.

📦 Jitter
A random offset added to the backoff delay, preventing a thundering herd of workers retrying at the exact same instant.

📦 Cap
An upper bound on wait time and/or attempt count, preventing an infinite retry loop.

📦 Circuit breaker
A protective mechanism that stops requests entirely once a dependency keeps failing, then probes it again over time.

Backoff without jitter means hundreds of jobs that failed at the same moment retry at the exact same millisecond — driving an already-struggling PSP into a worse state.

The backoff formula, and why a fixed wait isn't enough

A fixed one-second wait is simple but has two problems: if the PSP is passing through a brief spike, one second may not be enough; if the PSP has already recovered, one second is needless delay. Exponential backoff starts fast and grows more cautious with each attempt:

delay = min(cap, base * 2^attempt) + random(0, jitterRange)

attempt 0 → ~200ms
attempt 1 → ~400ms
attempt 2 → ~800ms
attempt 3 → ~1600ms
...
attempt N → hits the cap (e.g. 30s)

Without jitter, this formula is dangerous: every worker that failed at the same time retries at exactly 200ms, 400ms, 800ms later, hitting the PSP in synchronized waves. Adding a random amount (full jitter or decorrelated jitter) spreads that wave out.

The difference between retry and defer

Retry means the same worker, inside the same process, tries the same request again after a short wait — usually within seconds. Defer means putting the job back onto the database or queue to be picked up much later — minutes, sometimes hours. A rate-limited error is usually resolved with retry; but if the PSP itself is going through a large-scale outage, staying in a retry loop for minutes burns worker capacity and resources — deferring the job is the healthier way to 'put it to sleep' for a while.

Rate limited → retry (seconds, with backoff)
Extended PSP outage → defer (minutes, a separate scheduled re-attempt)

Circuit breaker: when to stop trying entirely

When requests to a dependency fail repeatedly, every new request does nothing but reproduce an already-known outcome — burning resources and inflating latency. A circuit breaker operates with three states:

Closed    → requests flow normally
  │ error threshold exceeded
  ▼
Open      → requests are rejected immediately, never reaching the PSP
  │ cool-down period elapses
  ▼
Half-Open → a limited number of probe requests are sent
  ├─ succeed → Closed
  └─ fail → Open

A circuit breaker doesn't replace retry; it's the layer above retry that catches, early, the moment retrying has become pure waste. While the breaker is open, workers should route jobs to the deferred queue rather than keep trying uselessly.

How many attempts, how high a cap

These numbers shouldn't be arbitrary; they should be proportional to the PSP's own SLA and the business value of the job. A high-value payment might warrant 8-10 attempts within a 5-minute window; a low-priority background job might need only 3.

Distinctions that get blurred

❌ Retry = defer
✓ Retry happens within seconds, in the same process; defer holds the job for minutes

❌ Jitter is a nice-to-have optimization
✓ Backoff without jitter makes the thundering-herd risk real, not theoretical

❌ A circuit breaker is an alternative to retry
✓ A circuit breaker is the layer that decides when to stop retrying

Full jitter vs. no jitter

Criterion	No jitter	Full jitter
Synchronized-wave risk	High	Low
Load pattern on the PSP	Sharp spikes	Spread out
Implementation complexity	Low	Slightly higher

Checklist for building the retry algorithm

1. Does the backoff formula have a cap, or could the wait grow unbounded in theory?
2. Is jitter applied, or do all workers retry at exactly the same instant?
3. Is there a real distinction between retry (rate limited) and defer (extended outage)?
4. Do workers actually stop sending requests to the PSP while the circuit breaker is open?
5. Was the attempt count and total window chosen based on the job's real business value, or picked arbitrarily?
6. Are the breaker's open/half-open/closed transitions tracked as a metric?

What to take away

1. Exponential backoff alone isn't enough; without jitter it produces synchronized waves of retries.
2. Retry and defer aren't the same verb: one happens in seconds, the other in minutes or hours.
3. A circuit breaker isn't an alternative to retry — it's the layer that catches, early, when retrying has turned into waste.
4. Attempt counts and caps should be chosen deliberately, based on real business value.

A good retry algorithm doesn't hide failure — it makes the cost of failure controlled.

The next part moves to the ground these retries actually run on: a database-backed job queue with leases, and how it prevents two workers from processing the same job at once.