Offset Management & Commit Strategies (commitSync vs commitAsync)

In Apache Kafka, consumers are entirely responsible for tracking their own read positions across topic partitions. The broker does not maintain per-message acknowledgement queues or delete messages when consumed.

Instead, consumers record their progress by committing Offsets to an internal, highly optimized Kafka topic named __consumer_offsets. How and when your application commits these offsets determines whether your pipeline achieves at-least-once delivery, at-most-once delivery, or suffers from silent data loss.

---

1. The Internal __consumer_offsets Topic

Offsets are stored as ordinary Kafka messages inside the internal topic __consumer_offsets (typically configured with 50 partitions and log compaction enabled):

• Group Coordinator Assignment: Each consumer group is assigned to a specific broker called the Group Coordinator. The coordinator is chosen by hashing the group.id: $\text{coordinatorBroker}=\text{Murmur2}(\text{group.id})(\mathrm{mod}50)$
• Log Compaction: Because consumers commit offsets continuously, older commits are superseded. Kafka's Log Cleaner daemon automatically purges outdated commits, keeping only the latest offset per [group, topic, partition] tuple.

---

2. The Danger of Automatic Offset Commits (enable.auto.commit=true)

By default, Kafka consumer clients enable auto-commits:

• enable.auto.commit = true
• auto.commit.interval.ms = 5000 ($5\text{ seconds}$)

When enabled, the consumer automatically commits the highest offset returned by the previous poll() call during the next poll() invocation if 5 seconds have elapsed.

Why Auto-Commit Causes Data Loss:

As illustrated above, auto-commit decouples offset persistence from actual business logic completion. If a crash occurs mid-batch, uncompleted records are permanently skipped upon restart.

---

3. Manual Commit Strategies: commitSync vs commitAsync

To guarantee at-least-once processing, you must disable auto-commit (enable.auto.commit = false) and commit offsets manually only after business logic succeeds.

A. commitSync(): Synchronous Blocking Commit

• The consumer thread blocks until the broker responds with an acknowledgement.
• Automatic Retries: If a transient network error occurs, commitSync() automatically retries until success or timeout.
• Performance Cost: Calling commitSync() after every single message reduces consumer throughput from $100,000\text{ msg/sec}$ to under $500\text{ msg/sec}$ due to network round-trip latency ($RTT$).

B. commitAsync(): Asynchronous Non-Blocking Commit

• Dispatches an OffsetCommitRequest over the network and immediately continues the processing loop without waiting.
• Why commitAsync() Does Not Retry: Suppose you commit Offset 200 asynchronously, but the request is delayed in the network. Meanwhile, you process the next batch and commit Offset 300 successfully. If the delayed commit for Offset 200 then fails and automatically retries, it could overwrite the newer Offset 300 with stale Offset 200, causing massive duplicate re-processing!

---

4. The Enterprise Standard: The Hybrid Async-Sync Commit Pattern

In production architectures, high throughput and zero data loss are achieved by combining commitAsync() during the steady-state poll loop with a guaranteed commitSync() inside the shutdown/rebalance lifecycle:

Production Implementation (TypeScript / KafkaJS):

---

5. Summary & Best Practices

1. Always set enable.auto.commit = false for any transactional, billing, or stateful pipeline.
2. Commit Offsets at Batch Boundaries: Commit once per batch rather than per record to eliminate network roundtrip bottlenecks.
3. Handle Idempotency Downstream: Because network retries and crash recovery commit with at-least-once semantics, ensure downstream databases use idempotent upserts or transactional inbox deduplication.