Apache Kafka Architecture: Topics, Partitions, Clusters, and Replication

In distributed streaming systems, Apache Kafka is engineered around a single core abstraction: the Distributed Partitioned Commit Log.

Unlike relational databases that organize data into mutable tables, or classical message brokers (RabbitMQ, SQS) that treat messages as ephemeral items to be deleted upon consumption, Kafka treats data as an infinite, ordered, immutable sequence of events.

To scale horizontally across hundreds of bare-metal machines or cloud instances while maintaining total ordering guarantees, Kafka decomposes the stream into Topics, shards topics into Partitions, distributes partitions across Brokers, and replicates partitions across failure domains.

---

1. Topics and Partitions: The Unit of Parallelism and Ordering

The Topic Abstraction

A Topic is a logical category or feed name to which records are published. In enterprise microservice architectures, topics represent distinct business domains or entity life cycles (e.g., orders.v1, telemetry.gps.raw, fraud.transactions.flagged).

However, a topic is strictly a logical concept. On the physical disk of a Kafka broker, a topic does not exist as a single file; instead, a topic is physically divided into one or more Partitions.

The Partition: Physical Log and Total Ordering Boundary

A Partition is a single, immutable, ordered append-only sequence of records that is continually appended to.

Invariant Rules of Kafka Partitions:

1. Monotonic Offsets: Each record within a partition is assigned a sequential 64-bit integer identifier called an Offset. Offsets are immutable and monotonically increasing within that partition ($0,1,2,3\dots$).
2. Strict Ordering Guarantee: Kafka guarantees total ordering within a single partition. However, Kafka provides NO ordering guarantees across different partitions of the same topic.
3. The Concurrency Axiom: The number of partitions in a topic defines the maximum degree of consumer parallelism. If a topic has $P=8$ partitions, a consumer group can have at most $8$ active consumer threads processing data concurrently. Any additional consumer threads in that group will remain completely idle.

---

2. Broker Nodes & Cluster Topologies

A Broker is a single Kafka server process running on a host machine or container. A Kafka Cluster is a cooperative collection of multiple broker nodes working together to provide distributed storage and high availability.

Physical Storage Directory Layout

On the host file system of a broker, partitions are represented as subdirectories within the configured data path (e.g., /var/lib/kafka/data):

Because each partition maps to an independent directory on disk, a broker can write to multiple partitions simultaneously utilizing multi-threaded sequential I/O across separate storage volumes (JBOD or RAID arrays).

---

3. Leader-Follower Replication and In-Sync Replicas (ISR)

To prevent data loss in the event of hardware failure, power loss, or network partitions, Kafka replicates each partition across multiple brokers.

The Replication Factor ($R$)

When creating a topic, you specify a Replication Factor $R$ (typically $R=3$ in production). This means every partition will have 1 Leader replica and $R-1$ Follower replicas placed on distinct broker nodes.

The Roles of Replicas:

1. Leader Replica:
   • One broker is designated the Leader for a given partition.
   • By default, all producer write requests and consumer read requests are handled exclusively by the Leader replica.
2. Follower Replicas:
   • Followers do not handle client write traffic. Instead, they act as internal consumers, issuing periodic FetchRequest RPCs to the Leader to pull new log records and append them to their local disk logs.
3. In-Sync Replicas (ISR):
   • The ISR is the dynamic set of replicas that are fully caught up with the Leader's log.
   • If a follower fails, experiences a long JVM stop-the-world GC pause, or falls behind by more than replica.lag.time.max.ms (default: 30,000ms), the Leader automatically evicts that follower from the ISR list.
4. The High Watermark (HW):
   • The High Watermark (HW) is the highest offset that has been replicated to all members of the ISR.
   • Consumers can only read up to the High Watermark. Records beyond the HW (which have been written to the leader but not yet replicated to the ISR) are hidden from consumers to prevent "dirty reads" in the event of an immediate leader crash.

---

4. Partition Sizing & Capacity Planning Formulas

Choosing the right number of partitions when provisioning topics is critical. Undersized topics choke consumer throughput, while massively oversized topics introduce metadata overhead and file descriptor exhaustion.

The Mathematical Sizing Formula:

Let:

• $T_p$: Target peak write throughput for the topic in $\text{MB/sec}$.
• $T_c$: Target peak consumption throughput for a single consumer thread in $\text{MB/sec}$.
• $S_{\text{partition}}$: Safe single-partition write throughput (typically $20\text{MB/sec}-50\text{MB/sec}$ on NVMe).
• $S_{\text{consumer}}$: Processing throughput of your slowest downstream consumer service (e.g., $5\text{MB/sec}$ due to database insertion latency).

The minimum required partition count $P$ is:

$P=\max \left(\lceil \frac{T_p}{S_{\text{partition}}}\rceil ,\lceil \frac{T_p}{S_{\text{consumer}}}\rceil \right)$

Example Calculation:

A fintech transaction topic must support a peak of $120\text{MB/sec}$.

• A single partition can ingest up to $30\text{MB/sec}$: $\lceil 120/30\rceil =4\text{ partitions}$.
• The downstream payment fraud detection microservice can only process $6\text{MB/sec}$ per container instance: $\lceil 120/6\rceil =20\text{ partitions}$.
• Therefore, the topic must be provisioned with at least $P=20\text{ partitions}$ to allow 20 fraud detection consumer pods to process the stream in real time without lag accumulation.

---

5. Production Failure Modes & Operational Gotchas

Failure Mode 1: The Partition Count Trap

• Symptom: A production topic is created with only 3 partitions. Months later, traffic surges 10x. Deploying 30 consumer instances results in 27 instances remaining permanently idle while 3 instances peg CPU at 100% and accumulate millions of lag messages.
• Why it happens: Kafka assigns each partition to exactly one consumer thread per consumer group. You cannot have more active consumers than partitions.
• The Gotcha: You can increase partition count dynamically (kafka-topics.sh --alter --partitions 20), but doing so changes the key hash modulo ($hash(key)(\mathrm{mod}P)$), breaking total ordering for future keyed messages!

Failure Mode 2: Unclean Leader Election (unclean.leader.election.enable)

• Scenario: The leader broker for a partition crashes. The only surviving follower brokers are out of sync (not in the ISR).
• If unclean.leader.election.enable = false (Production Default): The partition becomes unavailable for writes and reads until an ISR member recovers. Prioritizes Consistency ($C$) over Availability ($A$).
• If unclean.leader.election.enable = true: An out-of-sync follower is elected leader. All un-replicated records on the old leader are permanently lost, and consumer offsets become corrupted. Never enable this for financial, transactional, or audit workloads!

---

Summary Checklist

1. Logical vs Physical: Topics are logical categories; Partitions are the physical units of disk storage, ordering, and horizontal scaling.
2. Ordering Guarantee: Strict total ordering exists only within a single partition, never across partitions.
3. Replication Safety: Always configure production topics with replication.factor = 3, min.insync.replicas = 2, and producer acks = all.
4. Consumer Parallelism: Partition count sets the absolute ceiling on concurrent consumer processing threads.