Kafka KRaft Mode vs ZooKeeper: Architecture, Quorums, and Performance

For more than a decade, running an enterprise Apache Kafka cluster required operating two separate distributed systems in tandem: Apache Kafka (the streaming broker engine) and Apache ZooKeeper (an external centralized consensus service).

While ZooKeeper was instrumental in Kafka’s early success, its dual-system architecture imposed a severe ceiling on metadata scalability, partition count limits, controller failover recovery times, and operational complexity.

With the introduction of KIP-500 and KRaft (Kafka Raft Metadata Mode), Kafka completely eliminated its dependency on ZooKeeper, replacing external coordination with an internal, event-driven Raft consensus quorum.

---

1. The Bottlenecks of the Legacy ZooKeeper Architecture

To appreciate why KRaft is revolutionary, one must examine the fundamental failure modes of the legacy ZooKeeper model.

1. The Controller Sync Bottleneck & Asynchronous Lag

In a ZooKeeper-based cluster:

• One broker node is elected as the Kafka Controller.
• Metadata (topic configurations, partition assignments, ISR lists, broker registrations) was stored as hierarchical trees of ZNodes in ZooKeeper.
• Whenever a partition state changed (e.g., an ISR dropped a node), the Controller had to:
  1. Write the state update synchronously to ZooKeeper.
  2. Send individual LeaderAndIsr and UpdateMetadata RPC messages to every single broker in the cluster.

As clusters grew to hundreds of thousands of partitions, serializing and broadcasting these RPCs created massive latency bottlenecks.

2. Slow Controller Failover Times ($p99$ Latency Spikes)

When the active Controller broker crashed in a ZooKeeper cluster:

1. A new broker was elected Controller by creating an ephemeral ZNode in ZooKeeper.
2. The newly elected Controller had to read the entire cluster state from ZooKeeper from scratch—traversing millions of ZNodes to build its in-memory metadata cache.
3. During this initialization window (which took tens of seconds to multiple minutes on large clusters), partition reassignments, topic creations, and leader elections were completely frozen!

3. The 200,000 Partition Ceiling

Because ZooKeeper's memory footprint and watch notifications scaled poorly with fine-grained state updates, clusters were practically limited to approximately $200,000$ partitions before experiencing JVM GC pauses and metadata corruption risks.

---

2. The KRaft Paradigm: Treating Metadata as an Event Stream

KRaft (Kafka Raft) applies Kafka's core philosophy to its own control plane: Metadata is simply an append-only event stream.

Instead of storing metadata in an external tree, KRaft stores all cluster metadata in an internal, dedicated topic called @metadata.

The KRaft Node Roles:

In KRaft mode, nodes are configured with one of three roles (process.roles in server.properties):

1. controller: A dedicated node that participates in the Raft metadata quorum.
2. broker: A standard data node that hosts client partition logs and streams metadata from the controller quorum.
3. broker,controller (Combined Mode): A single node serving both roles, ideal for local development and lightweight staging clusters.

---

3. Sub-Second Failover & Instantaneous Controller State

In KRaft mode:

• All standby controllers in the Raft quorum continuously replicate and apply the @metadata log to their local memory in real time.
• When the active Controller Leader crashes:
  1. The standby controllers detect the missing Raft heartbeat and initiate a leader election.
  2. A new Controller Leader is elected in less than $100\text{ milliseconds}$.
  3. Because the new leader’s in-memory metadata table is already fully warm and up to date, it takes over immediately with zero initialization delay!

---

4. Metadata Snapshots (.checkpoint)

As cluster operations run for months, the @metadata log accumulates millions of records.

To prevent unbounded log growth and ensure new brokers can boot up instantly, KRaft periodically generates Metadata Snapshots:

When a brand-new broker joins the cluster:

1. It downloads the latest .checkpoint snapshot file, initializing its routing state in a single read.
2. It fetches only the few delta records appended after the snapshot offset.

---

5. Quantitative Benchmark: ZooKeeper vs KRaft

---

Summary Checklist

1. Self-Contained Consensus: KRaft replaces ZooKeeper with an internal Raft metadata quorum, drastically simplifying deployment and security architecture.
2. Metadata as an Event Stream: All cluster state changes are stored as immutable records in the internal @metadata topic.
3. Sub-Second Failover: Standby controllers maintain warm in-memory replicas of cluster metadata, enabling sub-$100\text{ms}$ leader failovers.
4. Massive Partition Scalability: Eliminates the legacy partition ceiling, enabling single clusters to scale to over 10 million partitions with consistent low latency.