CQRS (Command Query Responsibility Segregation) in Production: Consistency, Failure, and Recovery (CQRS In Production Consistency Failure And Recovery)

How these failures look to a user

If a banking balance is delayed by two seconds, a user may think money disappeared. If a like count is delayed by two seconds, most users may accept it. The same technical lag creates two different product risks.

Likewise, if the same OrderPlaced message arrives twice, inventory can be reduced twice. Production concepts should therefore be read not only as definitions, but through their effect on the user and the business.

The real production question

The happy path is simple: a command is accepted, an event is published, and a projection updates. In production, a message arrives twice, a consumer falls behind, a projection breaks, or a user cannot find the order they just created.

Command accepted → event published → projection delayed → user refreshes
"Did my order disappear?"

Partial failure is not an exception in distributed systems. This chapter is not about removing failure; it is about preserving data correctness, user trust, and recovery time when failure arrives.

Concepts at first use

📦 Consistency lag
The time from a successful write to an updated read model.

📦 Retry
A controlled repeat of work after a transient technical failure.

📦 Dead-letter queue (DLQ)
A queue that isolates messages the normal flow cannot process.

📦 Checkpoint
The last event position a projection has processed safely.

Eventual consistency does not mean data is wrong. It means different models reach the same fact at different times. If that is acceptable, the product must explain it clearly; if it is not, a critical query needs a different consistency strategy.

Every defence has a reason

• We keep a checkpoint because a replay needs to know where a projection can safely resume.
• We need idempotency because applying a duplicate event twice corrupts real business effects such as inventory or payment.
• We limit retry because repeating a malformed message cannot repair it; it can only amplify load and incorrect impact.
• We use a DLQ because a message unresolved by the normal flow needs visible ownership and investigation.
• The partition key is the aggregate ID because the causal order of one aggregate matters more than global event order.
• We replay because verified event history remains correct even when a projection is broken.

Consistency lag is a product decision

When a paid order does not appear in a list for two seconds, it can feel like data loss. Define the visible contract first: after an action, how quickly must each screen become current?

POST /orders → 202 Accepted + orderId + version
GET /orders/{id}?minVersion=42
  → current view when projection reaches 42
  → "processing" state while it has not

Optimistic UI, polling, or wait-for-version can help. They do not remove lag, because the real job is to explain its meaning honestly to the user. The command-side result remains the source of truth for critical decisions.

Duplicate events and reordering

At-least-once delivery makes duplicates normal. Processing one order event twice can reduce inventory twice. A consumer must check event identity and aggregate version.

if event.id already processed: ignore
if event.version <= projection.version: ignore
apply event
store checkpoint and event id

This is an O(1) average indexed lookup or O(log n) B-tree lookup. If ordering is guaranteed only inside a partition, use aggregate ID as the partition key, because the causal order of one order matters more than global order.

Retry, DLQ, and the operator decision point

Not every failure should be retried. A network timeout can be transient; an invalid event schema is permanent.

Failure type	Response	Why
Timeout / 503	Retry with exponential backoff	A dependency can recover
Rate limit	Delayed retry	Recover without amplifying load
Schema validation	DLQ + alert	Retrying cannot repair the message
Business-rule failure	Persist, inspect, compensate	Automatic retry can amplify wrong effects

A DLQ is not a bin. Every message needs an owner, review window, replay procedure, and alert.

How to recover a broken projection

A projection is not a cache; it is a rebuildable business view.

1. Stop the consumer or create a new projection version
2. Validate the last safe checkpoint
3. Replay the event stream in a controlled way
4. Validate counts and sample data
5. Route traffic to the new projection
6. Observe lag and error rate

Replay cost is linear O(n) in the number of events. Snapshots or segmented replay can reduce it, but a snapshot is not the source of truth. Because recovery must rely on verified event history.

The Saga failure path

When payment fails in e-commerce, inventory must not remain reserved forever. A saga is not a technical rollback; it is a business compensation.

Reserve inventory → Capture payment → Create shipment
                 payment fails → Release inventory

Compensation must be idempotent too: ReleaseInventory twice must not increase stock twice. Choreography stays light for local flows; orchestration gives a long workflow a state machine and one observation point.

Observability

Measure consumer lag, retry count, DLQ depth, checkpoint age, duplicate rate, and end-to-end action time. Carry a trace ID from command through event and consumer to the query returned to the user.

Mappings that create false confidence

❌ Retry = reliability
✓ Retry is safe only for transient errors and idempotent work.

❌ DLQ = recovery
✓ A DLQ starts an investigation and replay process.

❌ Replay = always safe
✓ Without versioning and isolated side effects, replay can repeat impact.

❌ Eventual consistency = random delay
✓ Lag must be measured, accepted with the product, and explained to users.

Production-readiness check

1. Do you have a target for user-visible consistency lag?
2. Is every consumer safe against duplicates and out-of-order events?
3. Does retry distinguish transient from permanent failure?
4. Does every DLQ message have an owner and replay runbook?
5. Can you rebuild a projection safely against production data?
6. Are Saga compensations idempotent?

If you cannot answer each question with evidence, the system may look scalable but is not yet operable.

What should remain with you?

1. Eventual consistency is a contract managed through user experience.
2. Duplicates and reordering are normal distributed delivery, not edge cases.
3. Retry, DLQ, and replay are one recovery system.
4. A projection that cannot be rebuilt becomes operational debt.
5. A saga does not roll back; it compensates business impact.

Production architecture appears not when messages arrive correctly on the first attempt, but in how the system behaves when they arrive twice, late, or in an unexpected order.