Kafka Storage Internals: Anatomy of Log Segments, .index, and .timeindex Files

At the lowest level of Apache Kafka's storage subsystem lies a simple yet formidable engineering principle: the partitioned commit log is not a single monolithic file on disk.

If a partition were stored as one unbounded file, operations such as garbage collecting expired data, replaying historical offsets, and locating specific records across terabytes of streaming data would require costly sequential scans and complex file truncation locks.

Instead, Kafka decomposes every partition into an ordered sequence of immutable disk units called Log Segments.

1. The Segment Lifecycle & Rollout Triggers

A partition directory contains multiple log segments. At any point in time:

• The Active Segment: Exactly one segment is currently open for appends. All incoming producer writes append sequentially to this segment's .log file.
• Closed Segments: All prior segments are closed and strictly immutable. They are read by consumers, memory-mapped by the kernel, or evaluated by background cleanup threads for deletion/compaction.

Segment Rollout Triggers:

Kafka rolls over the active segment and creates a new one whenever either of the following limits is breached:

1. Size-based Rollout (segment.bytes):
   • Default: 1,073,741,824 bytes ($1\text{ GB}$).
   • When the active .log file reaches $1\text{ GB}$, Kafka flushes the indexes, marks the segment closed, and opens a new segment whose base filename equals the next offset to be written.
2. Time-based Rollout (segment.ms):
   • Default: 604,800,000 ms ($7\text{ days}$).
   • For low-throughput topics that take weeks to accumulate $1\text{ GB}$, segment.ms forces a rollout so that time-based retention and compaction policies can purge old data without waiting indefinitely.

2. Binary Wire and Disk Format of Kafka Records

Kafka stores messages on disk in the exact same binary format used over the TCP network wire: RecordBatches containing individual Records.

The RecordBatch Header Components:

• BaseOffset (int64, 8 bytes): The absolute offset of the first record in this batch.
• BatchLength (int32, 4 bytes): Total size of the record batch in bytes.
• Attributes (int16, 2 bytes): Bitmask storing compression type (0: None, 1: Gzip, 2: Snappy, 3: LZ4, 4: Zstandard), timestamp type, and transactional markers.
• ProducerId (int64) & BaseSequence (int32): Used by Kafka's idempotent producer engine to transparently deduplicate retried network writes on the broker without disk rewrites.

3. The .index File: Sparse Indexing & Memory-Mapping

When a consumer asks to read records starting at offset $O=58,100$, scanning through a $1\text{ GB}$ binary .log file from byte $0$ would require megabytes of disk I/O and CPU parsing overhead.

To make offset lookups $\mathcal{O}(1)$ without consuming massive memory, Kafka creates a companion .index file for every log segment.

Why a Sparse Index Instead of a Dense Index?

• In a Dense Index, every single record has an entry in the index file. For 100 billion messages, index files alone would consume hundreds of gigabytes of RAM.
• In a Sparse Index, Kafka only writes an index entry every $N$ bytes of log data written (configured by index.interval.bytes, default: $4,096$ bytes / $4\text{ KB}$).
• An index entry consists of only 8 bytes:
  • relativeOffset (int32, 4 bytes): Calculated as $\text{Offset}-\text{BaseOffset}$ to save space.
  • position (int32, 4 bytes): The exact physical byte offset within the .log file where that batch begins.

The $\mathcal{O}(\log M)$ Lookup Algorithm:

When a consumer requests offset $O=54,206$:

1. Locate Segment: Kafka performs a binary search over the sorted list of segment base offsets in memory to find the segment file containing $54,206$ (in this case, segment 00000000000000054200).
2. Binary Search Sparse Index: Kafka performs a binary search over the memory-mapped .index file to find the largest indexed offset $\le 54,206$. It finds relative offset $+4$ (Offset $54,204$, at physical byte $4,096$).
3. Scan at most $4\text{ KB}$: The broker seeks directly to byte position $4,096$ in the .log file and scans sequentially through at most $4\text{ KB}$ of records to locate offset $54,206$.

By combining binary search on memory-mapped sparse indexes with micro-sequential disk scans, Kafka resolves offset lookups in microseconds with near-zero memory footprint!

4. The .timeindex File: Timestamp-to-Offset Translation

Modern streaming applications frequently need to rewind consumers to a specific point in time (e.g., "Replay events from yesterday at 14:00 UTC" via consumer.offsetsForTimes()).

To enable fast time-based lookups, Kafka maintains a .timeindex file:

1. A binary search over .timeindex finds the nearest relative offset matching the requested Unix timestamp.
2. That offset is then resolved via .index to locate the physical byte position in .log.

5. Log Compaction vs Log Deletion

Kafka provides two distinct cleanup strategies configured by cleanup.policy:

1. Delete Policy (cleanup.policy = delete):

• Closed segments older than retention.ms (default: 7 days) or exceeding retention.bytes are deleted in their entirety via a single OS unlink() syscall. Zero byte fragmentation.

2. Compact Policy (cleanup.policy = compact):

• Used for changelog streams and state tables (e.g., Kafka Streams KTable, database CDC streams).
• Background Log Cleaner threads retain the latest value for every key, discarding older superseded updates while preserving the original offsets.
• Deletions are executed by writing a message with a non-null key and a null payload (a Tombstone).

Summary Checklist

1. Segment Anatomy: Every partition is split into segment triads: .log (raw records), .index (offset-to-byte sparse index), and .timeindex (time-to-offset index).
2. Zero Ingestion Fragmentation: Messages are written to disk using the exact binary format used on the network wire.
3. Sparse Index Efficiency: Index entries are written every $4\text{ KB}$ (index.interval.bytes), enabling fast $\mathcal{O}(\log N)$ lookups with minuscule RAM requirements.
4. Cleanup Policies: delete drops old segment files atomically; compact deduplicates keys to maintain the latest state snapshot.