Multi-Cluster Kafka Replication: Active-Passive, Active-Active, and MirrorMaker 2

Enterprise systems operating at global scale cannot rely on a single Kafka cluster located in one cloud region. Data centers suffer power failures, underwater fiber cables get severed, and regulatory frameworks (like GDPR) demand regional data residency.

To build disaster-resilient and globally distributed event streams, organizations deploy multi-cluster architectures using MirrorMaker 2 (MM2).

---

1. Multi-Cluster Topologies: Active-Passive vs Active-Active

---

2. MirrorMaker 2 (MM2) Architecture

Built on top of the robust Kafka Connect distributed framework, MirrorMaker 2 coordinates replication using three dedicated connector components:

The Three MM2 Connectors:

1. MirrorSourceConnector: Reads records from the source cluster, preserves original timestamps and headers, and writes them to the destination cluster with an automatic Source Cluster Prefix (e.g., us-east.orders.v1).
2. MirrorCheckpointConnector: Replicates consumer group state and writes translation tables to checkpoints.internal, mapping source offsets to destination offsets.
3. MirrorHeartbeatConnector: Emits periodic heartbeats to track end-to-end network latency and cluster connectivity.

---

3. The Offset Translation Problem & Consumer Failover

A common trap in disaster recovery planning: Offsets in Cluster B DO NOT match offsets in Cluster A!

• Message at Offset 5000 in us-east.orders.v1 might be written to Offset 1240 in us-west.orders.v1 because timestamps, batching, and compaction differ between clusters.

---

4. Preventing Infinite Circular Replication Loops

In an Active-Active bidirectional setup, Cluster A replicates to Cluster B, and Cluster B replicates to Cluster A. Without proper safeguards, events replicate infinitely in an endless loop until the cluster crashes.

The Solution: Automatic Topic Namespace Prefixing

MM2 automatically renames replicated topics using the source cluster alias:

Because MM2 is configured with regex filters to ignore topics that already have a cluster prefix (topics.exclude = .*[\.-]internal, [a-zA-Z0-9_-]+\..*), circular replication loops are mathematically impossible.

---

5. Production MirrorMaker 2 Configuration (mm2.properties)

---

6. Summary Checklist

• Always configure sync.group.offsets.enabled = true to enable automated offset translation for disaster recovery failovers.
• Enforce strict topic naming prefixes to prevent infinite replication storms in Active-Active architectures.
• Test failover procedures regularly in staging using RemoteClusterUtils.translateOffsets() to verify that consumer groups resume accurately without duplicate storm processing.