Consumer Rebalancing: Eager vs Cooperative Sticky Protocol

In Apache Kafka, Rebalancing is the distributed consensus protocol used by a consumer group to dynamically distribute topic partitions among active member instances whenever members join, leave, or crash, or when topic partition counts change.

Historically, Kafka's legacy rebalancing mechanism triggered catastrophic "Stop-The-World" pauses across entire consumer fleets. Modern Kafka (v2.4+) introduces Incremental Cooperative Rebalancing (KIP-429) and Static Membership (KIP-345) to achieve zero-downtime rolling deployments.

1. The Rebalancing Lifecycle: Group Coordinator & Protocol Wire Frames

Every consumer group is managed by a specific broker designated as the Group Coordinator.

2. The Legacy Eager Rebalance Protocol: "Stop-The-World" Pathology

Under the legacy Eager Rebalance Protocol (RangeAssignor, RoundRobinAssignor):

1. As soon as a rebalance is triggered, every single consumer instance must revoke ALL its assigned partitions.
2. All data consumption halts across the entire group.
3. Every consumer sends a JoinGroup request and waits for the coordinator.
4. After assignments are redistributed, consumers reconnect and resume fetching.

The Rolling Update "Rebalance Storm"

If a Kubernetes cluster performs a rolling deployment of 20 consumer pods one by one:

• Pod 1 shuts down $\to$ Rebalance 1 (Full Cluster Halt)
• Pod 1 starts up $\to$ Rebalance 2 (Full Cluster Halt)
• Pod 2 shuts down $\to$ Rebalance 3 (Full Cluster Halt)
• ...
• Total: 40 consecutive cluster-wide stops, resulting in minutes of total consumer downtime, massive consumer lag spikes, and downstream SLA breaches.

3. Incremental Cooperative Rebalancing (KIP-429)

Introduced in Kafka 2.4, the Cooperative Rebalancing Protocol replaces the eager all-or-nothing model with a two-phase non-blocking migration:

Key Differences:

• Unaffected Partitions Never Pause: If Consumer 2 owns Partition 2 and Partition 3, and a new node joins taking Partition 1 from Consumer 1, Consumer 2's streaming pipeline is never interrupted.
• Preserves Local State & Caches: In Kafka Streams or stateful consumers with in-memory RocksDB state stores, sticky reassignment prevents expensive state restore downloads over the network.

4. Static Group Membership (KIP-345): Surviving Transient Restarts

In containerized environments (Kubernetes, AWS ECS), pods frequently restart due to node maintenance, config updates, or crash-loop recoveries.

Under dynamic membership, a restarting pod generates a new member.id (e.g., client-1-7e834b9d), triggering an immediate rebalance.

The Solution: group.instance.id

By providing a persistent Static Instance ID (such as the Kubernetes Pod name):

• group.instance.id = "payment-consumer-pod-0"
• session.timeout.ms = 45000 ($45\text{ seconds}$)

5. Landmark Global Arena Capstone #4 Integration

In this module's connected Landmark Arena Capstone, global-kafka-cooperative-sticky-assignor, you will build the core partition assignment engine for an enterprise cooperative assignor:

1. Uniform Balance Constraint: Ensure no consumer has more than $\lceil \frac{N}{M}\rceil$ partitions, where $N$ is total partitions and $M$ is active consumers.
2. Stickiness Maximization: Minimize partition migrations from previous assignment states.
3. Cooperative 2-Phase Revocation Output: Explicitly return partitions to retain, partitions to revoke, and partitions to newly assign.

6. Production Configuration Profile