Designing Event-Driven Systems With Guaranteed Delivery

You want “guaranteed delivery” in event-driven systems because you have scars. A payment got captured, but the “order-confirmed” event never arrived. A user got emailed twice. A downstream job ran, crashed mid-write, then ran again and quietly corrupted data. These are not edge cases; they are the default failure modes of distributed systems.

Here is the plain definition you can use in design reviews: guaranteed delivery means your system can survive crashes, network failures, and retries without losing events, and without producing incorrect outcomes when duplicates occur. In practice, that usually means at-least-once delivery paired with idempotent processing, and occasionally it means leveraging exactly-once features within a tightly controlled boundary.

The uncomfortable truth is this: the queue is not the reliable part. Reliability emerges from how your database writes, your event publishing, your consumer logic, and your retry behavior fit together. If any one of those is sloppy, “guaranteed delivery” is marketing language, not engineering.

What experts actually say about “exactly once.”

People who have built large-scale event systems tend to agree on one thing: exactly-once delivery is a useful abstraction, not a promise you can make end-to-end.

Jay Kreps, co-creator of Apache Kafka, has repeatedly emphasized that stronger delivery semantics come from aligning producer writes, broker persistence, and consumer progress tracking. The breakthrough was not a magic broker flag, but carefully coordinated state transitions across components. Those guarantees only hold when applications use the transactional APIs correctly and stay within defined boundaries.

Martin Kleppmann, distributed systems researcher and author, consistently frames the problem more bluntly. Failures cause retries, retries cause duplicates, and once you interact with external systems, exactly-once becomes an illusion. The only durable solution is to make state transitions deterministic and idempotent so reprocessing does not change the outcome.

Cloud messaging platforms reinforce this reality in their defaults. Redelivery is not a bug; it is a feature. If a consumer does not explicitly acknowledge successful processing, the system assumes failure and retries. That is a design statement, not an implementation detail.

Taken together, the expert consensus is clear: guaranteed delivery is built from durable writes, retries, deduplication, and visibility into failure, not from trusting the broker alone.

The three delivery semantics you are actually choosing from

Most event-driven systems collapse into three practical options:

1. At-most-once
   Fast and lossy. Messages may be dropped. Acceptable for telemetry, unacceptable for money.

2. At-least-once
   Messages are not lost, but duplicates are possible. This is the foundation of most real “guaranteed delivery” systems.

3. Exactly-once (within a boundary)
   Achievable in limited contexts when the platform tightly controls state transitions. Once you cross service, database, or API boundaries, duplicates return.

A useful mental model is this: brokers guarantee storage and redelivery rules, applications guarantee correctness.

A comparison table that saves architectural arguments

If your architecture includes a database plus an event bus, the transactional outbox pattern eliminates more production incidents than any broker feature.

Build guaranteed delivery the boring way

If correctness matters more than elegance, this is the path that works.

Step 1: Eliminate dual writes with a transactional outbox

The most common failure pattern is simple. Your service writes to its database, then publishes an event. The database commit succeeds, the publish fails, and your system state diverges from reality.

The transactional outbox pattern fixes this by storing the event in an outbox table within the same database transaction as the business change. A separate relay process later publishes those events to the broker.

A clean baseline looks like this:

• Insert the business record.
• Insert an outbox record containing event ID, type, and payload.
• Commit once.
• Let a relay safely publish and retry.

If the business state exists, the event exists. There is no window where reality can split.

Step 2: Make consumers idempotent with a processed-events ledger

At-least-once delivery guarantees duplicates. That is not hypothetical; it is statistically inevitable.

The standard consumer pattern is:

• Each event carries a globally unique event ID.
• The consumer records that ID in a processed-events table as part of the same transaction as the side effect.
• If the event ID already exists, the handler exits without reapplying changes.

This pattern turns at-least-once delivery into effectively-once outcomes. Without it, retries will eventually corrupt state.

Step 3: Acknowledge messages only after durable work completes

Acknowledgements are where teams accidentally destroy reliability.

The rule is simple and unforgiving:

• Only acknowledge after the side effect is committed durably.

• Crashing before acknowledgement must be safe.

• Retries must be intentional, visible, and bounded.

If you write one operational checklist, it should include:

• Exponential backoff for transient failures
• Dead-letter queues after a fixed retry limit
• Alerts on dead-letter growth
• Replay tooling for operators
• Structured logs with event ID and correlation ID

That checklist is the difference between “guaranteed delivery” and endless incident reviews.

Step 4: Use broker features to reduce pain, not responsibility

Broker guarantees can help, but they do not absolve application design.

If you use Kafka-style transactional processing, it excels at pipelines that stay entirely within the broker ecosystem. Once you write to an external database or call a third-party API, you are back in the land of retries and idempotence.

FIFO queues can dramatically reduce duplicates, but they trade throughput and flexibility for stronger guarantees. They work best when message ordering and deduplication windows match your workload.

Ack-based systems give you control, but also put the burden on you to get timing and failure handling right.

In all cases, broker features reduce the frequency of duplicates, not the need to handle them safely.

A worked example that explains why “duplicates are rare” is wrong

Assume a checkout system publishes 10,000 events per minute, about 167 per second. One in a thousand succeeds but times out waiting for confirmation, so it gets retried.

That is a retry rate of 0.1 percent.

• Events per minute: 10,000
• Duplicate rate: 0.1 percent
• Duplicates per minute: 10
• Duplicates per day: 14,400

That is not theoretical noise. That is a guaranteed source of bugs unless your consumers are idempotent and observable.

FAQ

Is end-to-end exactly-once delivery possible?

Sometimes within tightly controlled boundaries. Across databases, services, and external APIs, you design for at-least-once and neutralize duplicates.

Should I use two-phase commit?

Rarely. It multiplies failure modes and operational complexity. Transactional outbox patterns solve the same problem with fewer sharp edges.

How do I deduplicate safely?

Use stable event IDs, enforce uniqueness at the data layer, and apply side effects and dedupe recording in the same transaction.

What should I monitor?

Outbox backlog, publish retries, consumer retries, dead-letter rates, dedupe hits, and end-to-end latency from event creation to effect completion.

Honest Takeaway

Guaranteed delivery is not a feature you enable. It is a discipline you enforce.

The most reliable systems assume retries, accept duplicates, and make those duplicates harmless. Transactional outboxes, idempotent consumers, and disciplined acknowledgements are not glamorous, but they work.

Once that foundation is in place, broker features can make life easier. They cannot make reality simpler.