Message Delivery Guarantees: At-Most-Once, At-Least-Once, and Exactly-Once

In distributed systems, the fundamental reality of unreliable networks and independent process failures means that message delivery is never free of tradeoffs.

Whenever a producer sends a record to a message broker, or a consumer reads and processes that record, network packets can be lost, broker leaders can crash, and database transactions can time out.

To design resilient architectures, software engineers must understand the three core delivery guarantees provided by messaging infrastructure:

1. At-Most-Once Delivery: Messages may be lost, but are never duplicated.
2. At-Least-Once Delivery: Messages are guaranteed never to be lost, but may be delivered more than once.
3. Exactly-Once Semantics (EOS): Each message is processed effectively once, producing deterministic state changes across producers, stream processors, and storage sinks.

---

1. At-Most-Once Delivery (Fire-and-Forget)

Under At-Most-Once delivery, the system prioritizes maximum throughput and minimal latency over reliability. In the event of a crash or network partition, messages are dropped rather than retried.

How to Configure At-Most-Once:

• Producer Configuration:
  • acks = 0: The producer sends the record over the network socket and immediately returns success without waiting for an acknowledgment from the broker.
  • retries = 0: The producer never retries failed network transmissions.
• Consumer Configuration:
  • enable.auto.commit = true: The consumer commits the offset to __consumer_offsets as soon as records are fetched from poll(), before the application business logic executes. If the consumer crashes halfway through processing, the offset has already advanced.

When to Use:

• Real-time IoT sensor telemetry where dropping $1$ reading out of $1,000$ does not impact aggregate analytics.
• High-volume clickstream logs and performance trace metrics.

---

2. At-Least-Once Delivery (Retries + Acknowledgment)

Under At-Least-Once delivery, the system guarantees that no message is ever lost. However, because network acknowledgments can drop, consumers may receive the same message multiple times.

How to Configure At-Least-Once:

• Producer Configuration:
  • acks = all (-1): The broker leader only acknowledges the produce request after all in-sync replicas (ISRs) have written the record to their local write-ahead log.
  • retries = Integer.MAX_VALUE: The producer will retry indefinitely upon transient network blips or leader re-elections.
  • min.insync.replicas = 2: Guarantees that at least 2 replicas successfully write the record before acknowledging.
• Consumer Configuration:
  • enable.auto.commit = false: The consumer manually commits offsets only after the business logic and database transaction have successfully completed.

The Inevitability of Duplicates:

If a consumer processes a batch, updates the database, and then crashes before committing its offset to Kafka, the re-assigned consumer will re-read the exact same records. At-Least-Once delivery requires Consumer-Side Idempotency to prevent duplicate side effects.

---

3. Exactly-Once Semantics (EOS)

Achieving true Exactly-Once Semantics (EOS) across a distributed streaming pipeline requires coordination across three distinct boundaries:

1. Idempotent Producer: Eliminates duplicate writes from producer retries.
2. Transactional Stream Processing (Read-Committed): Atomic read-process-write loops across Kafka topics.
3. Sink Database Commit: Transactional bridge between Kafka and external databases.

A. How the Idempotent Producer Works (enable.idempotence = true)

When idempotence is enabled on the producer:

1. The Kafka cluster assigns the producer a unique 64-bit Producer ID (PID) via an InitProducerId request.
2. For each topic partition, the producer maintains a monotonically increasing Sequence Number ($0,1,2,\dots$) for every message sent.
3. The broker leader caches the latest 5 sequence numbers received from each PID.
4. If the producer sends sequence number $4$ and the network drops the ACK, the producer re-sends sequence number $4$.
5. The broker inspects its sequence cache, detects that sequence number $4$ was already written to the log, and returns a successful ACK to the producer without writing a duplicate record.

$\text{Duplicate Drop Condition: }\text{incomingSeqNum}\le \text{lastCommittedSeqNum}$

---

B. Transactional Coordinator and Two-Phase Commit (2PC)

For stream-processing applications (e.g., Kafka Streams, Apache Flink) that read from Topic A, update an internal aggregate, and write to Topic B:

• Kafka introduces a Transaction Coordinator and a dedicated internal topic __transaction_state.
• The consumer offset commit and the outgoing topic produce request are bound together in a single atomic transaction.
• The broker writes a Commit Marker to the partition log upon completion. Downstream consumers configured with isolation.level = "read_committed" only see records whose transactions have successfully committed.

---

4. The "End-to-End Fallacy": Why EOS Breaks at the Database Sink

A frequent architectural misconception is that enabling Kafka's isolation.level = "read_committed" guarantees that an external database (PostgreSQL, MongoDB, Stripe API) will never receive duplicate operations.

Why This is Physically Impossible Without Sink Idempotency:

Kafka's Transaction Coordinator can only coordinate commits inside the Kafka cluster. It has no visibility or two-phase commit lock on your PostgreSQL database or external payment gateway.

The Golden Rule of Distributed Delivery:

$\text{End-to-End Exactly-Once}=\text{At-Least-Once Delivery}+\text{Consumer Idempotency}$

---

5. Failure Mode Deep-Dive: The Double-Charge Credit Card Incident

The most dangerous failure in event-driven systems is executing non-idempotent third-party API mutations inside an At-Least-Once consumer loop.

The Incident Sequence:

1. Consumer reads PaymentInitiated event for order_100.
2. Consumer calls Stripe API: POST https://api.stripe.com/v1/charges without an Idempotency-Key header.
3. Stripe receives the request, processes the credit card transaction, and deducts \500.00$.
4. A transient network timeout occurs before Stripe's HTTP 200 response reaches the consumer.
5. The consumer catches a SocketTimeoutException and retries the handler.
6. The consumer issues a second POST https://api.stripe.com/v1/charges without an idempotency key.
7. Stripe charges the customer \500.00$ a second time.

---

6. Code Deep-Dive: Idempotent Consumer Pipeline with PostgreSQL Deduplication

Below is a complete, production-ready TypeScript implementation of an Idempotent Kafka Consumer that safely processes financial transactions under At-Least-Once delivery using a database-level idempotency key:

---

7. Comparative Delivery Guarantees Matrix

---

Summary and Key Takeaways

1. At-Most-Once delivery sacrifices durability for speed, committing offsets prior to execution and dropping messages during failures.
2. At-Least-Once delivery guarantees zero data loss, retrying unacknowledged writes and committing offsets only after successful processing.
3. Kafka's Idempotent Producer prevents producer-side duplicate writes using unique Producer IDs (PIDs) and monotonically increasing sequence numbers.
4. Kafka EOS coordinates transactions strictly within the Kafka cluster; external databases and HTTP APIs require sink-level idempotency keys to prevent duplicate mutations.
5. The universal standard for mission-critical enterprise systems is At-Least-Once delivery combined with atomic database deduplication (ON CONFLICT DO NOTHING).