A customer is charged twice for one order. The payment worker did its job: it read the message, charged the card, and was about to tell the queue “done”. Then the process crashed. The queue never heard the acknowledgement, assumed the message was lost, and sent it to another worker. The second worker charged the card again.
Nothing was broken. This is the system working as designed, and it is the most important idea in messaging. This lesson shows what the guarantees mean, why “exactly once” needs a second ingredient, and how to make duplicates harmless.
- Kafka vs RabbitMQ: a queue deletes a message once it is acknowledged, a log keeps it.
- Event-driven architecture, for why services send messages at all.
When does RabbitMQ consider a message handled? When the consumer acknowledges it. Kafka does the equivalent by saving the consumer’s position in the log. This lesson is about the moment between doing the work and reporting it.
- A broker can promise at-most-once or at-least-once. It cannot promise that a message is processed exactly once.
- The order of “process” and “acknowledge” decides the guarantee.
- Any consumer on an at-least-once system must expect duplicates.
- An idempotent consumer, usually a dedupe table with a unique constraint, turns at-least-once into effectively-once.
- Retries need a limit and a dead-letter queue, or one bad message blocks everything.
Say which delivery guarantee a design gives and why, make a consumer safe against duplicates, and explain why exactly-once delivery is really at-least-once plus idempotency.
The Three Guarantees#
Kafka’s documentation lists the options plainly. In its message delivery semantics section:
| Guarantee | What it promises |
|---|---|
| At most once | messages may be lost but are never redelivered |
| At least once | messages are never lost but may be redelivered |
| Exactly once | each message is processed once and only once |
A plain letter dropped in a box may vanish and is never sent twice: at most once. A parcel that needs a signature is sent again until someone signs: at least once. Exactly once would mean the parcel arrives once even if the courier’s van breaks down mid-route, and nobody can promise that.
The documentation splits the problem in two: durability when publishing and guarantees when consuming. Out of the box, Kafka guarantees at-least-once; you get at-most-once only by disabling producer retries and saving the consumer’s position (its offset) before processing.
One Crash, Two Outcomes#
On the consuming side, a consumer does two things: it processes a message and it acknowledges it. Only the order differs, and the order picks the guarantee.
Kafka’s documentation describes the same two choices for a consumer that saves its position. Save first, then process, and a crash in between means the replacement starts after messages that were never processed: at-most-once. Process first, then save, and the first few messages the replacement receives were already processed: at-least-once.
A consumer processes a message, then the machine loses power before it acknowledges. What does the broker do?
It cannot know the work happened, so it delivers the message again. RabbitMQ’s documentation says so directly: when manual acknowledgements are used, any delivery that was not acked is automatically requeued when the channel or connection closes, and consumers must be prepared to handle redeliveries and be implemented with idempotence in mind.
RabbitMQ’s automatic acknowledgement mode counts a message as delivered once it is sent. The documentation says it is unsafe and not suitable for all workloads: at-most-once in practice.
Cloud queues use a timer instead of a connection#
Amazon SQS has no open channel to watch, so it uses a visibility timeout. A received message is hidden, not removed, and the consumer deletes it when done. If it is not deleted in time, it becomes visible again and another consumer can take it. The default is 30 seconds.
Set the timeout shorter than your slowest job and two workers run the same message at once. Even inside the window, standard queues are at-least-once, and AWS tells you to use idempotent operations.
The publishing side#
The same uncertainty exists when sending, and each broker answers it differently:
- Kafka. After a network error, a producer cannot tell whether the message was committed, so it resends: at-least-once. The idempotent producer (since 0.11.0.0) lets the broker discard the repeats. In the 4.3 docs
enable.idempotencedefaults to true unless conflicting settings turn it off. - RabbitMQ. A publisher that has written a message to its socket cannot assume it reached the server, so the broker sends a publisher confirm, which mimics the consumer acknowledgement.
- SQS FIFO. A send that is retried with the same deduplication ID within 5 minutes is not added twice. Note what that covers: duplicate sends, inside a window. A consumer that crashes before deleting still sees the message again.
Saving and sending: the dual write#
One more gap sits before the broker. A service saves an order, then publishes “order placed”. If it crashes between the two, the order exists and nobody hears about it. Publishing first has the opposite failure.
The fix is the transactional outbox. The service writes the message into an outbox table in the same transaction as the order, and a separate relay publishes from there.
The outbox pattern write-up names the catch: a relay that crashes after publishing but before marking the row sent publishes it again. The outbox removes lost messages, not duplicates, so the consumer still has to be idempotent.
”Exactly Once” Is a Claim to Read Carefully#
Kafka’s documentation is blunt: many systems claim exactly-once delivery, but read the fine print, because the claims can mislead when consumers or producers fail, when there are multiple consumer processes, or when data written to disk is lost.
What real systems offer is effectively once: deliver at least once, then arrange that a repeat has no extra effect. There are three ways.
- Idempotent producer. Kafka’s producer can ask the broker to assign it an ID and deduplicate messages using a sequence number, so resending does not create duplicate entries.
- Commit the result and the position together. For a Kafka-to-Kafka pipeline, the consumer’s offset is written in the same transaction as the output, so either both happen or neither does. Readers must set
isolation.level=read_committed; the default,read_uncommitted, also shows aborted writes. - Idempotent processing. For everything else, the work itself must be safe to repeat.
For an external system such as a database or a payment provider, the documentation says exactly-once delivery generally requires cooperation with the destination, for example storing the offset with the output. In plain terms: you have to make the effect idempotent.
Making a Consumer Idempotent#
An operation is idempotent if repeating it changes nothing the second time.
| Operation | Safe to repeat? |
|---|---|
| set the order status to “paid” | yes |
| upsert a row by primary key | yes |
| add 10 to a balance | no |
| charge a card, send an email | no |
Some work is naturally safe. Kafka’s documentation points out that when messages have a primary key, updates are idempotent, because receiving the same message twice just overwrites a record with another copy of itself. Charging a card or sending an email is not safe, so those need a guard.
The usual guard is a dedupe table: record each message ID, with a unique constraint, in the same transaction as the work.
on message(m):
BEGIN
INSERT processed(m.id)
-- fails if id seen
UPDATE orders SET ...
COMMIT
ack(m)If the insert fails because the ID exists, the consumer rolls back, acknowledges, and skips the work. A crash after the commit but before the ack is now safe: the redelivery hits the unique constraint and does nothing. The same constraint also covers two workers holding the same message at once, as after an expired visibility timeout: only one insert wins.
The ID must come from the sender (an order ID, a request ID), not be made up per delivery, or every redelivery looks new. And the ID and the effect must commit together, or a crash between them brings the problem back.
When the effect is outside your database#
A card charge happens at the payment provider, so your transaction cannot roll it back. Here the dedupe has to live where the effect lives: pass the sender’s ID to the provider as an idempotency key. Stripe’s API, for example, returns the saved result when a request is retried with the same Idempotency-Key, and errors if the parameters differ.
Keys are up to 255 characters, and every dedupe store forgets eventually. A repeat that arrives after that is treated as new:
| Dedupe store | Remembers an ID for |
|---|---|
| Stripe idempotency keys | may be pruned after 24 hours |
| SQS FIFO send dedupe | 5 minutes |
| your own dedupe table | as long as you keep the rows |
Keep IDs for longer than a redelivery can arrive, including a replay from the dead-letter queue days later.
Retries, Poison Messages and Dead Letters#
Sometimes a message fails every time: bad data, a bug, a missing record. Requeueing it at once creates a loop. RabbitMQ’s documentation warns that such a requeue and redelivery loop can be costly in network bandwidth and CPU, and suggests tracking redeliveries and either rejecting for good or requeueing after a delay.
A dead-letter queue is where those messages go. Each broker decides when to move them:
- RabbitMQ dead-letters a message when it is rejected with requeue set to false, expires, or overflows a queue length limit.
- RabbitMQ quorum queues also dead-letter a message returned more times than the delivery limit, which defaults to 20 since 4.0. With no dead-letter target set, the message is dropped. Since 4.3 the count ignores explicit nacks that set
delivery_failed=false. - SQS uses a redrive policy: after
maxReceiveCountreceives without a delete, the message moves to the dead-letter queue.
A sensible policy has three parts:
- Retry a few times with growing, slightly random delays (backoff with jitter), so many failing consumers do not retry in lockstep.
- Then dead-letter the message so the queue keeps moving.
- Alert and replay. Someone looks, fixes the cause, and sends the messages back through.
Retries also break order: a message that fails comes back after later ones. The table below says what to do.
Choosing a Guarantee#
| Workload | Reasonable choice |
|---|---|
| Metrics, logs, telemetry you can lose | at-most-once, simplest and fastest |
| Orders, payments, emails | at-least-once plus an idempotent consumer |
| Kafka-to-Kafka processing | transactions: commit output and offset together |
| Save a row, then publish an event | outbox, plus an idempotent consumer |
At-least-once with idempotency is the default answer for anything that matters. Be wary of any design that depends on “exactly once” with no idempotent step you can point to.
Interview Follow-ups#
Ask first whether a message can be lost, whether the effect can repeat, and whether order matters per key. State your assumption aloud. The companion lesson Message Ordering, Rebalances, Transactions and Sagas answers the rest of the follow-ups.
| Probe | Short answer |
|---|---|
| Ordering at scale? | Use the entity ID as the key, so one entity’s events share a partition. Retries still reorder. |
| Consumer group rebalance? | A consumer too slow to poll is treated as failed and its work is repeated. The dedupe check absorbs it. |
| Several services must agree? | No single transaction spans them. The companion covers sagas, Kafka transactions and outbox relays. |
Mid: names at-least-once and idempotency. Senior: adds the dedupe transaction, external keys, DLQ policy and rebalance duplicates. Staff: also covers dedupe retention, replay, outbox or sagas across services, and what to monitor. These are this guide’s judgement, not a published rubric.
Where AI Systems Hit This#
Agents retry too. A model that calls a refund tool, times out, and calls it again is the same problem: two deliveries, one intended effect. Give the action a stable ID and make the tool ignore repeats. The agent interview lesson uses this example, and the guardrails lesson covers checks around tool calls.
- Saying “the queue guarantees exactly-once” with no idempotent step.
- Generating a new ID on each retry, which defeats deduplication.
- Retrying forever with no limit and no dead-letter queue.
- Saving the message ID and doing the work in two separate steps.
- Writing to the database and publishing in two steps, with no outbox.
Try It: Count the Duplicates#
A payment consumer uses RabbitMQ with manual acknowledgements and a prefetch of 50, which limits how many unacknowledged messages it can hold. Assume it crashes 3 times a day, and each time it holds a full 50 unacknowledged messages that it has already processed. Decide:
- What is the most redelivered messages per day?
- What happens to the customers if the charge is not idempotent?
- What one change fixes it?
Hint
Multiply crashes by messages held at a crash. Then think about what a redelivered, already-processed payment message does.
One answer
Per crash: up to 50 repeats
3 crashes/day: up to 150
Each repeat is a charge that
was already done, so up to
150 double charges a day
Fix: send the order id as
the provider's idempotency
key on every attemptRedelivery after a crash is normal, so the fix cannot be “avoid crashes”. The charge happens at the provider, outside your transaction, so the order ID goes to it as the idempotency key. A local dedupe table protects your own database. Lowering the prefetch shrinks the worst case but does not remove it.
When asked “can you guarantee exactly-once?”, do not say yes. Say: “The broker gives at-least-once. A dedupe table keyed by the order ID, committed with the effect, makes redeliveries harmless, and payment calls carry the same ID as an idempotency key. Messages that keep failing go to a dead-letter queue after a few retries.” Then name the limit: a third party with no idempotency key, or a dedupe store whose IDs expired, can still see a repeat.
Check Yourself#
1.A consumer acknowledges a message and then crashes before finishing the work. Which guarantee is this, and what is the cost?
At-most-once. The message is acknowledged, so it is never redelivered, and the unfinished work is lost.
2.Why is a dedupe ID generated by the consumer on each delivery useless?
Each redelivery would get a new ID, so every repeat looks like new work. The ID must come from the sender and stay the same across retries.
3.You must send exactly one welcome email per signup, and the queue is at-least-once. What do you do?
The send is outside your database, so no single transaction covers it. Use the email provider’s idempotency key if it has one, keyed by the signup ID. If it does not, record the signup ID before sending and skip any repeat; a crash between the record and the send can then lose one email, so pick which failure you prefer and say so.
4.What would change if this consumer read from SQS and a job sometimes took longer than the visibility timeout?
The message would become visible again while the first worker is still busy, so a second worker runs it at the same time, with no crash at all. Raise the timeout above the slowest job or extend it while working, and keep the dedupe check: the unique constraint lets only one of the two commit.
5.One message fails every time and blocks the queue. What is the fix?
Limit retries, with growing delays, then move the message to a dead-letter queue and alert, so the rest of the queue keeps moving and someone can fix and replay it.
Draw receive, process, acknowledge from memory. Mark where a crash loses the message and where it duplicates it, then add the dedupe check. Repeat tomorrow.
In One Picture#
At most ack first, may lose
At least ack last, may repeat
Exactly at least once +
idempotent work
Dedupe sender id, unique key,
same transaction as the effect
External idempotency key, keep
ids past the retry window
Publish outbox; relay repeats
Failures few retries, then DLQDeliver at least once, make the work idempotent, and send the stubborn failures to a dead-letter queue.
- The brokers behind this: Kafka vs RabbitMQ covers how each one stores and acknowledges messages.
- Why services send messages at all: event-driven architecture.
- The same idea for agents: AI agent interview questions uses the double-refund example.
Next in the course: design a URL shortener, the first full design in the classic problems module.
