Kafka Consumer Groups Explained: How Multiple Consumers Share Workloads

In Apache Kafka, Consumer Groups are the primary mechanism for horizontal scalability, fault tolerance, and load distribution on the consumption side.

While traditional message queues (such as RabbitMQ or AWS SQS) track individual message acknowledgments at the broker and dispatch messages randomly to any listening worker, Kafka coordinates consumption at the partition level.

---

1. The Cardinal Rule of Kafka Consumption

The fundamental invariant governing Kafka consumer groups is:

Each topic partition can be actively read by at most one consumer instance within a given Consumer Group at any point in time.

Key Takeaways from the Invariant:

1. Competing Consumers within a Group: Instances within the same group.id share the workload. Partitions are divided among members so that every message is processed exactly once per group.
2. Independent Fan-Out across Groups: Multiple consumer groups with distinct group.id values read from the same topic independently at their own pace without affecting each other's offsets.
3. Deterministic Partition Ordering: Because only one thread within a group reads Partition 0, records within Partition 0 are processed in strict sequential order.

---

2. Consumer Scaling Dynamics: Partitions as the Ceiling of Parallelism

The number of partitions in a topic represents the absolute maximum concurrency limit for any single consumer group.

[!IMPORTANT] If your topic has 8 partitions, spinning up a 16-pod Kubernetes deployment for your consumer group will leave 8 pods permanently idle with 0 CPU utilization. To increase consumer parallelism, you must first increase the topic partition count.

---

3. Partition Assignment Strategies (Assignors)

How partitions are assigned to consumer group members is determined by the partition.assignment.strategy configuration. Kafka provides four standard assignor implementations:

---

A. RangeAssignor (org.apache.kafka.clients.consumer.RangeAssignor)

• How it Works: Evaluates each topic individually. For each topic, it arranges partitions in numerical order and consumers in lexicographical order, dividing the number of partitions by the number of consumers.
• The Math: Let $N=\text{partitions}$, $M=\text{consumers}$. The first $N(\mathrm{mod}M)$ consumers get $\lceil N/M\rceil$ partitions; the rest get $\lfloor N/M\rfloor$.

The RangeAssignor Skew Antipattern:

If a consumer group subscribes to 10 topics, each with 3 partitions ($N=3$), across 2 consumers ($M=2$):

• Consumer 1 gets Partition 0 & 1 for all 10 topics (Total: 20 partitions).
• Consumer 2 gets Partition 2 for all 10 topics (Total: 10 partitions).
• Result: Consumer 1 handles $200\%$ more load than Consumer 2, creating an artificial bottleneck.

---

B. RoundRobinAssignor (org.apache.kafka.clients.consumer.RoundRobinAssignor)

• How it Works: Flattens all partitions across all subscribed topics into a single list and distributes them across consumers in a round-robin cycle.
• Result: Solves the multi-topic skew problem of RangeAssignor. Across the same 10 topics (30 total partitions), Consumer 1 and Consumer 2 each receive exactly 15 partitions.

---

C. StickyAssignor (org.apache.kafka.clients.consumer.StickyAssignor)

• How it Works: Balances partition distribution as uniformly as possible while attempting to preserve existing assignments across rebalances.
• Why Sticky Matters: When a single consumer pod restarts, traditional assignors shuffle all partitions across all consumers. Sticky assignors reassign only the partitions owned by the departing node, keeping all other consumers bound to their current partitions and preserving local in-memory caches.

---

D. CooperativeStickyAssignor (Modern Standard - KIP-429)

• How it Works: Implements the Incremental Cooperative Rebalancing Protocol.
• Rather than revoking all partitions during a rebalance (which halts all consumption across the group), consumers continue processing unaffected partitions seamlessly. Only migrating partitions are temporarily paused during reassignment.

---

4. Production Consumer Group Configuration

---

5. Summary & Best Practices

1. Size Partition Counts for Future Consumer Scale: Topics cannot be consumed with higher parallelism than their partition count.
2. Use CooperativeStickyAssignor: Avoid the default RangeAssignor on multi-topic consumer groups to eliminate partition assignment skew.
3. Monitor Consumer Group Lag via JMX: Track records-lag-max across all consumer groups to identify lagging consumers before SLAs are breached.