What is a Distributed Commit Log? How Apache Kafka and Redpanda Work

To solve the memory exhaustion, destructive consumption, and throughput limits of traditional message queues, LinkedIn engineers (led by Jay Kreps, Neha Narkhede, and Jun Rao) introduced a radical paradigm shift in 2011: The Distributed Commit Log.

Instead of treating messages as transient tasks to be tracked, leased, and destroyed by the broker, Apache Kafka treats event data as an immutable, append-only, ordered log of records persisted to disk.

Understanding the mechanics of the distributed commit log—how partitions scale, how consumers track their own offsets, and how retention policies govern log storage—is the single most important foundation in modern event-driven and streaming architectures.

---

1. The Anatomy of an Append-Only Commit Log

At its lowest level, a commit log is the simplest possible storage abstraction: an ordered sequence of record entries written strictly to the end of a physical file (append-only).

A. The Inversion of Broker State: The "Dumb Broker, Smart Consumer" Model

In traditional queues (like RabbitMQ), the broker maintains complex data structures tracking which consumers hold unacknowledged leases on which messages.

In a distributed commit log, the broker does zero per-message state tracking:

1. Immutable Storage: Once written to a partition, a record is never modified or individually deleted.
2. Deterministic Offsets: Every record within a partition is assigned a monotonically increasing 64-bit integer called an Offset ($0,1,2,3,\dots$).
3. Consumer-Managed Positions: Each consumer group is responsible for tracking its own position in the log by periodically saving its current offset integer to an internal topic named __consumer_offsets.
4. Zero Overhead for Multiple Readers: Whether 1 consumer or 1,000 independent consumer groups read from a partition, the broker's workload is identical: performing sequential disk reads from the OS PageCache.

$\text{Broker Overhead Per Message}=\mathcal{O}(1)$

---

2. Partitions: The Fundamental Unit of Parallelism and Ordering

A single physical log file on a single server cannot scale infinitely. To scale storage and throughput horizontally across dozens of machines, Kafka subdivides every Topic into one or more Partitions.

The Rules of Kafka Partitions:

1. Total Order is Guaranteed Within a Partition Only:

   • Records within Partition 0 are strictly ordered by offset ($0\to 1\to 2\to 3$).
   • There is no global ordering guarantee across different partitions. Offset 3 in Partition 0 might have been written before or after Offset 3 in Partition 1.

2. Partition Key Hashing:

   • When a producer sends a record with a key, Kafka computes the target partition using the formula:

   $\text{Partition}=\text{Murmur2}(\text{serializedKey})(\mathrm{mod}\text{TotalPartitions})$

   • As long as the topic's partition count does not change, all records with the same key (e.g., customerId: "cust_9921") are guaranteed to land on the exact same partition in exact chronological order.

3. Single Consumer Per Partition Within a Consumer Group:

   • Within a single consumer group, a partition can be consumed by at most one consumer instance at any given time.
   • If a topic has 8 partitions, you can run up to 8 active consumer instances in parallel. Adding a 9th consumer instance will leave it completely idle as a hot standby.

---

3. High-Throughput Physics: Why Commit Logs Outperform Databases

How can Kafka write and serve millions of messages per second per broker while persisting every single byte to disk? It leverages three fundamental operating system and hardware physics principles:

1. Sequential Disk I/O vs Random Disk I/O

Magnetic HDDs and NVMe SSDs are optimized for sequential block streaming.

• Random disk writes (typical in B-Trees / relational databases) require constant pointer lookups and disk arm seeks, capping performance around $100-1,000\text{ IOPS}$.
• Sequential disk writes (Kafka's append-only model) can stream at the physical limit of the bus: $600\text{ MB/sec}$ on standard SATA and $>3.5\text{ GB/sec}$ on NVMe SSDs, outperforming random writes to main memory.

2. The Linux PageCache

Kafka does not manage its own in-process memory cache in Java heap memory (avoiding JVM garbage collection pauses). Instead, all disk I/O routes through the OS PageCache:

• When a producer writes a message, it is written directly to the OS kernel page cache and marked dirty.
• When a consumer reads that message milliseconds later, the kernel serves the bytes directly from RAM in the PageCache without touching the physical storage disk.

3. Zero-Copy Network Transfer (sendfile)

In traditional architectures, transferring data from disk to a network socket requires 4 context switches and 3 memory copies:

$\text{Disk}\stackrel{\text{DMA}}{\to }\text{OS PageCache}\stackrel{\text{CPU}}{\to }\text{JVM User Space}\stackrel{\text{CPU}}{\to }\text{Socket Buffer}\stackrel{\text{DMA}}{\to }\text{NIC}$

Kafka utilizes the Linux sendfile(2) system call (Zero-Copy):

$\text{Disk}\stackrel{\text{DMA}}{\to }\text{OS PageCache}\stackrel{\text{Direct DMA}}{\to }\text{Network Interface Card (NIC)}$

The data bypasses user space entirely, eliminating CPU copy overhead and context switching.

---

4. Log Retention Strategies: Time-Based vs Log Compaction

Unlike traditional message queues where messages are deleted upon ACK, Kafka retains records according to explicit retention policies.

A. Time-Based and Size-Based Retention (cleanup.policy=delete)

• retention.ms: Retain records for a fixed duration (e.g., 604800000 = 7 days). Once a segment file's latest record timestamp is older than 7 days, the entire segment is scheduled for disk deletion.
• retention.bytes: Limits the maximum disk footprint per partition (e.g., 107374182400 = 100 GB).

B. Log Compaction (cleanup.policy=compact)

In a compacted topic, Kafka retains the latest value for every primary key indefinitely.

• Tombstones: To delete a key in a compacted topic, a producer publishes a record with that key and a null payload (a Tombstone). During compaction, the key is permanently removed.
• Use Case: Recreating in-memory database caches, tracking user profile states, or capturing CDC table states.

---

5. Failure Mode Deep-Dive: The Premature Retention Pruning Outage

One of the most catastrophic silent failures in Kafka production environments is Premature Log Segment Truncation.

The Outage Scenario:

1. Topic telemetry.raw.events is configured with retention.ms = 86400000 (24-hour retention).
2. The primary analytics data warehouse consumer crashes on Friday at 6:00 PM due to an unhandled schema evolution exception.
3. Because it is the weekend, on-call alerts are acknowledged but the fix is not deployed until Monday at 10:00 AM (64 hours later).
4. Meanwhile, the Kafka broker continues purging log segments older than 24 hours.
5. When the analytics consumer is restarted, its committed offset ($offset=5,120,000$) no longer exists on disk (the oldest available offset on the broker is now $offset=9,450,000$).
6. The consumer encounters an OffsetOutOfRangeException.
7. If configured with auto.offset.reset = "latest", the consumer silently jumps to the tail, permanently skipping and losing 40 hours of critical financial data without raising an application error.

---

6. Code Deep-Dive: Time-Travel Offset Seeking & Historical Backfilling

Because Kafka preserves the commit log, consumers can programmatically rewind or fast-forward their position to reprocess past events or recover from outages.

Below is a production-grade TypeScript implementation using kafkajs demonstrating how to programmatically seek to an exact historical timestamp to backfill corrupted data:

---

7. Architectural Comparison: RabbitMQ vs Apache Kafka vs Redpanda

---

Summary and Key Takeaways

1. A distributed commit log is an immutable, append-only file on disk, where records are indexed by sequential 64-bit offsets ($0,1,2,\dots$).
2. Kafka scales horizontally via Partitions, providing total ordering within a single partition and routing keyed records deterministically via $\text{Murmur2}(\text{key})(\mathrm{mod}N)$.
3. Consumers track their own offsets, enabling multiple independent services to read at different speeds, rewind to historical timestamps, and replay data without impacting broker performance.
4. Kafka achieves extreme throughput using OS PageCache sequential I/O and Zero-Copy network transfers, bypassing JVM user-space memory entirely.
5. In the next lesson, we will explore the mechanics of Pull-Based vs Push-Based Consumption Models, analyzing how Kafka prevents consumer memory exhaustion via natural backpressure.