The payment consumer is idempotent, the outbox is in place, and the dead-letter queue is wired up. Then the interviewer asks the questions that decide the level: what keeps one customer’s events in order, what happens when the group rebalances, and how do three services agree on one outcome?
These are the follow-ups to Message Delivery Guarantees and Idempotency. They share one habit: each answer names a failure that repeats or reorders work even when nothing crashes.
- Message Delivery Guarantees and Idempotency: at-least-once delivery, dedupe tables and the outbox.
- Kafka vs RabbitMQ: partitions, consumer groups and offsets.
Why is a dedupe ID generated on each delivery useless? Every redelivery would look new. The ID must come from the sender. This lesson adds the cases where the repeat comes from the system itself, not a crash.
- Order is a per-key promise: the same key goes to the same partition, and retries can still reorder it.
- A consumer group rebalance repeats uncommitted work with no crash, so the dedupe check stays on.
- Kafka transactions cover Kafka to Kafka only. A database or payment API outside Kafka needs its own idempotent write.
- An outbox relay that polls and one that reads the database log can both publish a row twice.
- A saga replaces one cross-service transaction with local steps and compensating steps, and every step must be safe to repeat.
Answer the ordering, rebalance, transaction, relay and saga follow-ups on a messaging design in a few sentences each, and name the failure each answer still leaves open.
Which Broker, and What Order Does It Keep?#
Ask the same four questions first: can a message be lost, can the effect be repeated, does order matter per key, and how many messages arrive per second? Say your assumption aloud. Then the broker answers differ mainly in how they order and acknowledge.
| Broker | Ordering and acknowledgement |
|---|---|
| Kafka | A replayable log, ordered within a partition |
| SQS | FIFO queues order within a message group; standard queues are best effort |
| RabbitMQ | A queue with acknowledgements and a prefetch limit |
| Pulsar | Individual or cumulative acknowledgements; its Key_Shared mode keeps per-key order |
Kafka’s design page describes a topic as a set of totally ordered partitions, each consumed by exactly one consumer in a group at any time. So the usual answer is to pick the entity ID as the message key. One order’s events then share a partition and arrive in sequence.
Global order costs all parallelism, because only one partition and one consumer could keep it. A retry on the consumer side also reorders: a message that fails returns after later ones, so either block that key until it succeeds or make handlers tolerate it.
On the producer side, retries are safe by default. enable.idempotence defaults to true, and the producer docs say order is preserved with idempotence on, even with up to 5 requests in flight. Turn it off and max.in.flight.requests.per.connection above 1 can reorder after a failed send.
The Duplicate With No Crash: Rebalances#
A Kafka consumer must keep polling. One that does not poll within max.poll.interval.ms, which is 5 minutes by default, is treated as failed, and the group moves its partitions to another consumer.
A consumer is alive and healthy, but one batch takes longer than the poll interval. How many times does that batch run?
At least twice. The new owner fetches the last committed offset and repeats everything after it, while the slow consumer may still be finishing. Two defences work together: keep batches small enough to finish well inside the interval, and keep the dedupe check, whose unique constraint lets only one of the two runs commit.
Kafka also has a newer consumer group protocol, chosen with group.protocol; the default is still classic. It changes who coordinates the group, not the rule that a consumer which stops polling is considered failed, so the batch-size defence stays.
The same page describes static membership, where group members keep fixed IDs so that a restart does not trigger a rebalance. Judgement: it helps for planned restarts, but it does not fix a handler that is simply too slow.
What Kafka Transactions Cover#
For a pipeline that reads from Kafka and writes back to Kafka, a transaction makes the output records and the consumer offsets commit atomically. The producer needs a transactional.id. The consumer needs isolation.level=read_committed and enable.auto.commit=false.
When a transaction ends, marker records are written to show whether it committed or aborted, and read_committed readers skip the aborted records. Setting transactional.id also means a restarted application aborts any in-flight transaction of its previous instance.
Two limits matter in an interview:
- A stuck transaction is aborted. The coordinator aborts it after
transaction.timeout.ms, which is 1 minute by default. - The guarantee stops at Kafka’s edge. A database write or a payment call is outside the transaction, so it still needs an idempotent write or a provider idempotency key, as in the parent lesson.
Kafka 4.2 also made share groups production-ready, with per-record acknowledgements and no partition pinning. Judgement: use them for independent tasks, as with a work queue, and not for streams that need per-key order.
Backpressure#
Backpressure looks different per broker. Kafka is pull-based: a slow consumer falls behind and catches up later, so the signal to watch is consumer lag, together with dead-letter queue depth. RabbitMQ bounds the work in flight with prefetch.
Outbox Relay: Polling or Change Data Capture?#
The relay is the part of the outbox that reads the table and publishes. The transactional outbox pattern stores the message in the database first, so messages are sent if and only if the transaction commits, without two-phase commit. Its page also warns that the relay might publish a message more than once, so consumers must be idempotent. Two relay designs exist, and microservices.io lists the drawbacks of each.
| Relay | How it works | Drawback listed |
|---|---|---|
| Polling publisher | Polls the outbox table | ”Tricky to publish events in order” |
| Log tailing | Reads the database transaction log | ”Tricky to avoid duplicate publishing” |
Log tailing is what change data capture tools do. Debezium’s PostgreSQL connector shows the pattern:
- It records the log position of each event and resumes from it after a restart.
- After a crash its replacement tasks might repeat recent events, and the docs advise consumers to expect duplicates.
- Its outbox event router puts an event
idin the message headers, which it says can be used to remove duplicate messages.
Judgement: log tailing avoids a polling loop, but it is another system to run and monitor. Whichever relay you pick, the consumer stays idempotent.
Several Services, One Outcome: Sagas#
Placing an order may touch payment, stock and shipping, each with its own database. No single transaction spans them. The saga pattern defines a saga as a reliably coordinated sequence of local transactions. If a step fails, the changes made by the completed steps are undone by compensating transactions.
It names two ways to coordinate:
- Choreography. Each local transaction publishes events that trigger the next one.
- Orchestration. An orchestrator tells participants which local transaction to run.
The page also lists the costs: the developer must design the undo steps, and there is a lack of isolation, so concurrent sagas can expose anomalies.
Judgement: because the messaging underneath is at-least-once, every step and every compensation must be idempotent. A refund that runs twice is as bad as a charge that runs twice.
What Dies, and What Happens#
| What dies | What happens |
|---|---|
| Consumer, before ack | Redelivery. Dedupe absorbs it. |
| Consumer, after ack, before finishing | Work lost (at-most-once). |
| Consumer, too slow to poll | Rebalance. The batch is repeated; dedupe absorbs it. |
| Producer, before the confirm | It resends. The idempotent producer or dedupe ID absorbs it. |
| Outbox relay, after publish | Row published again. Consumer dedupes. |
| Dedupe database | Stop consuming. Never process without the check. |
| Broker node | Judgement: with replicas, writes acknowledged by all in-sync replicas survive one failure, but can still be lost if the rest fail too. |
- Promising global order when only per-key order is cheap.
- Forgetting that a slow consumer is treated as a dead one.
- Claiming Kafka transactions make a database write exactly-once.
- Writing compensating steps that are not idempotent.
Try It: Size the Batch#
Assume a Kafka consumer reads batches of 500 messages and each message takes 0.8 seconds to process. The poll interval is the 5-minute default. Decide:
- How long does one batch take, and does it fit?
- What does the group do?
- What two changes fix it?
Hint
Multiply messages by seconds per message, then compare with 300 seconds.
One answer
500 x 0.8 s = 400 s
Limit: 5 min = 300 s
400 s > 300 s: too slow
Group calls it failed and
moves the partition. Next
owner repeats the batch.
Fix: batch of 250 = 200 s
Keep the dedupe check tooThe batch overruns the interval, so the group treats a healthy consumer as failed and repeats the whole batch. A smaller batch fits with margin. The dedupe check still stays, because a slow disk or a pause can overrun even a smaller batch.
Check Yourself#
1.Why choose the order ID as the message key?
Kafka keeps order within a partition, and the same key goes to the same partition, so one order’s events stay in sequence while different orders run in parallel.
2.A healthy consumer takes 6 minutes on a batch. What does the group do, and what do you change?
With the default interval of 5 minutes it counts the consumer as failed and moves its partitions, so the batch is repeated. Shrink the batch or raise the interval, and keep the dedupe check.
3.Your pipeline reads Kafka and writes to Postgres. Do Kafka transactions make it exactly-once?
No. The transaction covers Kafka outputs and offsets only. Make the Postgres write idempotent, for example with a dedupe row committed in the same database transaction.
4.What would change if the outbox relay read the database log instead of polling?
No polling loop and a different ordering story, but the relay can still publish a row twice after a restart, so the consumer stays idempotent.
Pick one entity, say an order. Trace its events through key, partition, rebalance and relay, and mark every place a repeat or a reorder can appear.
In One Picture#
Order key = entity id, one
partition, per-key only
Rebalance slow = dead, batch
repeats, so keep dedupe
Kafka txn Kafka to Kafka only
Relay poll or log tail,
both can repeat
Saga local steps + undo,
all idempotentOrder per key, expect repeats from rebalances and relays, and make every step safe to repeat.
- The core lesson: Message Delivery Guarantees and Idempotency covers acknowledgements, dedupe tables and dead-letter queues.
- The brokers behind this: Kafka vs RabbitMQ.
Then move on to the classic problems with design a URL shortener.
