Lesson 7.2• 8 min read

Delivery guarantees, ordering and idempotency

At-most-once, at-least-once, exactly-once semantics, and deduplication with idempotency keys.

What you will learn

  • Tell at-most-once, at-least-once and exactly-once apart, and why exactly-once is about processing, not delivery.
  • Make a consumer idempotent by recording message IDs in the same transaction as its work.
  • Keep order where it matters by partitioning on an entity key.

The spectrum of distributed delivery semantics

In any distributed network where nodes, switches, and processes can fail, message transport guarantees fall into three categories:

  1. At most once: Messages may be lost but are never redelivered.
    • For example, the producer does not wait for the broker to confirm, or the consumer records its position before doing the work. If something fails at the wrong moment, the message is lost.
  2. At least once: Messages are never lost but may be redelivered.
    • The broker keeps the message until the consumer confirms it has been processed; if that confirmation is lost or times out, the message is delivered again, so the work can happen twice.
  3. Exactly once: Each message is processed once and only once.
    • A sender cannot tell a lost message from a lost acknowledgment, so over an unreliable network it must either risk losing the message (at most once) or risk sending it twice (at least once). "Exactly once" is therefore achieved in processing, not delivery.
    • Inside Kafka this is supported by an idempotent producer and transactions that write to several partitions atomically. When the output goes to an external system, the consumer must store its position in the same place as its output (or make the processing idempotent), so a retry cannot apply the same message twice.

Designing idempotent consumers with idempotency keys

An operation is idempotent if applying it multiple times produces the exact same system state as applying it once:

f(f(x))=f(x)f(f(x)) = f(x)what its symbols mean

where ffwhat its symbols mean is the operation and xxwhat its symbols mean the state it is applied to.

Implementation: record processed message IDs in the same transaction

The reliable guard is the database itself. Record each message's ID in a table whose primary key is that ID, inside the same transaction as the business update. If the message was already processed, the insert fails and the whole transaction rolls back, so the update is never applied twice:

async function processPaymentEvent(event: PaymentEvent) {
  const { messageId, userId, amount } = event;
  try {
    await db.transaction(async (tx) => {
      // Fails with a unique-key violation if this message was already processed.
      await tx.insert('processed_messages', { message_id: messageId });
      await tx.update('accounts').where({ id: userId }).decrement('balance', amount);
    });
  } catch (err) {
    if (isUniqueViolation(err)) return { status: 'DUPLICATE_SKIPPED' }; // already done: ignore
    throw err; // a real failure: let the broker redeliver
  }
  return { status: 'PROCESSED' };
}

A fast check in Redis (SET key value NX) can sit in front of this, but it must not be the only guard: if the consumer crashes after setting the Redis key and before the database commit, every retry would be skipped and the payment never applied.


Message ordering hazards: Global vs partition ordering

A frequent misconception in system design is requiring global FIFO ordering across all system events. Global FIFO requires a single monolithic queue with concurrency N=1N=1what its symbols mean, causing an extreme performance bottleneck.

  • Solution: Enforce causal ordering by entity key. A customer's banking actions (Deposit →\rightarrow Transfer →\rightarrow Balance Check) must execute sequentially. But Customer A's actions are completely independent of Customer B's actions.
  • By setting PartitionKey = userId, all events for a single user land on the same partition and process in order, while many partitions are processed in parallel across the worker cluster.

Delivery guarantees, retries and idempotency keys

At-least-once delivery, idempotent consumer: The consumer records each message ID in the same database transaction as the charge. A redelivered message's insert fails, the transaction rolls back, and the message is skipped.
Consumer account balance
$1000.00
Ledger (2)
Order payment #9021
Message ID: msg_101
-$50.00
10:14:02
Order payment #9022
Message ID: msg_102
-$120.00
10:14:15
Trace
System ready. Consumer listening on topic "payments.charges.v1"
Payment requests1× instanceLoad balancer2× instancesPayment producers2× instancesBroker1× instanceBilling consumers4× instancesFast check1× instanceLedger database1× instancePayment requests1× instanceLoad balancer2× instancesPayment producers2× instancesBroker1× instanceBilling consumers4× instancesFast check1× instanceLedger database1× instance
Figure 2

An idempotent consumer, with message IDs recorded in the ledger database.

Follow a payment message: the broker delivers it to a consumer, which records the message’s ID and applies the payment in one database transaction. If the consumer’s confirmation is lost, the broker delivers the message again; the second insert of the same ID fails, the transaction rolls back, and the payment is not applied twice. Delivery is at least once; processing is exactly once.

  • Payment requests: 1 × Users
  • Load balancer: 2 × Load balancer
  • Payment producers: 2 × App server
  • Broker: 1 × Queue
  • Billing consumers: 4 × App server
  • Fast check: 1 × Cache
  • Ledger database: 1 × Relational database
  • Payment requests sends requests to Load balancer
  • Load balancer sends requests to Payment producers
  • Payment producers sends requests to Broker
  • Broker sends requests to Billing consumers
  • Billing consumers sends requests to Fast check
  • Billing consumers sends requests to Ledger database

Key takeaways

  • A sender cannot tell a lost message from a lost acknowledgment, so it must risk losing messages or sending them twice.
  • With at-least-once delivery, an idempotent consumer makes redelivery harmless: the insert of a seen ID fails and the transaction rolls back.
  • A Redis check can speed this up but must not be the only guard, or a crash between the check and the commit skips the work for good.
  • Global ordering forces one consumer; order by entity key (for example userId) and process many partitions in parallel.
Sources4