Kafka Capacity Planning and Partition Reassignment: Adding Brokers Safely

As data volumes grow, streaming architects must scale Apache Kafka clusters horizontally by provisioning new broker nodes.

However, a critical operational reality often catches teams off guard:

When you add new broker nodes to an existing Kafka cluster, Kafka DOES NOT automatically move existing partitions onto the new nodes.

Newly added brokers sit completely idle until new topics are created or you explicitly execute a Throttled Partition Reassignment Plan.

---

1. Mathematical Capacity Planning Model for Kafka

Before provisioning hardware, use this mathematical model to size CPU, Network, and Disk storage:

Worked Capacity Example:

• Target Ingress: $100\text{ MB/sec}$ continuous write traffic.
• Replication Factor: $RF=3$.
• Retention Period: $7\text{ days}=604,800\text{ seconds}$.
• Active Consumer Groups: $4$ independent consumer groups.

1. Storage Calculation:

$\text{Storage Required}=100\text{ MB/s}\times 3\times 604,800\text{ s}\times 1.30=235,872,000\text{ MB}\approx 235.8\text{ TB}$

2. Network Egress per Broker Node (Assuming 6 Broker Cluster):

• Inter-broker follower replication: $100\text{ MB/s}\times 2=200\text{ MB/s}$.
• Consumer reads: $100\text{ MB/s}\times 4=400\text{ MB/s}$.
• Total Cluster Network Egress: $600\text{ MB/s}$ ($4.8\text{ Gbps}$).
• Per-Broker Network Requirement: $\frac{4.8\text{ Gbps}}{6}=800\text{ Mbps}$ (Provision $10\text{ Gbps}$ NICs).

---

2. The Step-by-Step Partition Reassignment Runbook

When expanding a cluster from 3 brokers (101, 102, 103) to 5 brokers (101, 102, 103, 104, 105):

---

Step 1: Create Topics JSON Input (topics-to-move.json)

---

Step 2: Generate Reassignment Plan

Run the CLI tool specifying the target broker list:

---

Step 3: Execute Reassignment with Network Throttling (CRITICAL)

[!CAUTION] If you run partition reassignment without --throttle, the brokers will replicate historical data at maximum unthrottled link speed ($10\text{ Gbps}$). This will saturate broker network interfaces and disk I/O, causing massive request timeout spikes across active production applications.

Apply a strict bandwidth cap (e.g., $50\text{ MB/sec}=52,428,800\text{ bytes/sec}$):

---

Step 4: Verify and Decommission Throttling

Periodically poll status until migration completes:

Once all partitions report Reassignment completed successfully, remove the throttle so ongoing production replication is not capped:

---

3. Summary & Best Practices

1. Never Execute Unthrottled Reassignments: Always cap reassignment bandwidth to no more than $20\%-30\%$ of available NIC capacity.
2. Execute During Low-Traffic Windows: Schedule partition moves during off-peak hours to minimize impact on real-time consumer SLAs.
3. Verify Throttle Removal: Ensure dynamic broker throttle configs are explicitly cleared after reassignment completes.