The Kafka Idempotent Producer: Producer IDs, Sequence Numbers, and Deduplication

In distributed systems, the network is inherently unreliable. When an application publishes a record to a message broker, three things can happen:

1. The message and acknowledgement both succeed.
2. The message is lost in transit (no write occurs).
3. The write succeeds on the broker, but the acknowledgement is dropped on the return network path.

In case 3, a standard producer retries the request, leading to duplicate message writes and out-of-order records.

The Kafka Idempotent Producer (introduced in KIP-98 and enabled by default in Kafka 3.0+) guarantees that network retries produce zero duplicate records and strict sequential order per partition.

---

1. The Anatomy of Network Retries & Duplication

---

2. How the Idempotent Producer Works Internally

The Idempotent Producer solves duplication at the storage engine level using two internal identifiers: Producer ID (PID) and Monotonic Sequence Numbers.

The Invariants:

1. Producer ID (PID): A unique 64-bit identifier assigned to the producer client instance by the cluster during connection initialization.
2. Sequence Number (seq): A 0-indexed integer assigned to every ProducerBatch for a specific TopicPartition.
3. Broker Deduplication Rule: When a broker receives a batch with PID $P$ and Sequence $S$:
   • If $S=\text{last\_seen\_seq}+1$: Accept & Append to Log. Update $\text{last\_seen\_seq}=S$.
   • If $S\le \text{last\_seen\_seq}$: Duplicate Detected! Acknowledge the batch to the producer without writing a duplicate record to disk.
   • If $S>\text{last\_seen\_seq}+1$: Out-of-Order Gap Detected! Reject with OutOfOrderSequenceException to prevent out-of-order writes.

---

3. Preserving Message Ordering with Pipelining (max.in.flight.requests = 5)

In non-idempotent producers, preserving in-order delivery required setting max.in.flight.requests.per.connection = 1. This destroyed network throughput because the producer had to wait for an ACK after every single socket write before sending the next frame.

Why Idempotence Enables Safe Pipelining:

Because the broker tracks sequence numbers in memory, even if Batch 2 arrives at the network socket before Batch 1 (due to TCP retransmissions or multi-connection routing), the broker buffers Batch 2 and refuses to commit it until Batch 1 (Seq 0) arrives, guaranteeing strict partition order while sustaining multi-gigabit pipelined throughput.

---

4. Multi-Language Configuration: Enabling Idempotence

A. Java

B. Go (Franz-Go)

---

5. Summary & Key Invariants

1. Idempotence is Scoped to a Single Producer Session: If the producer process crashes and restarts, it receives a new PID. Cross-session and cross-topic atomicity requires Kafka Transactions.
2. Zero Performance Cost: Broker deduplication is performed entirely in RAM using ring buffers; it adds zero disk read overhead.
3. Never Disable Idempotence: Disabling idempotence reintroduces silent duplicate writes on any transient network blip.