Single-Leader Replication Architecture: WAL Shipping, Failover, and Split-Brain

In production systems, deploying a single-leader database is only half the battle. You must design for the inevitable failure of the leader node without corrupting data or causing catastrophic Split-Brain divergence.

This lesson explores how database engines replicate mutations on the byte level, how automated failover pipelines promote standbys, and how Fencing Tokens protect distributed storage from zombie leaders.

---

1. How Database Engines Ship Changes

A leader database can broadcast its state mutations to followers using three fundamentally distinct log formats:

A. Statement-Based Replication

The leader forwards every mutating SQL statement (INSERT, UPDATE, DELETE) across the network. Followers parse and re-execute each SQL query.

• Critical Flaws:
  • Non-deterministic functions produce inconsistent state (e.g. NOW(), RAND(), UUID()).
  • Queries with side effects, triggers, or autoincrement sequences execute out-of-sync unless strictly serialized.

B. Write-Ahead Log (WAL) Shipping (Physical Replication)

The leader writes every disk-page-level byte modification sequentially to its Write-Ahead Log before updating data files. It streams these raw binary disk blocks (WAL segments) directly to replicas.

• Advantages: Ultra-fast, zero SQL parsing overhead, 100% exact bit-for-bit byte copy of the database storage engine.
• Disadvantages: Couples replicas tightly to the exact database engine version, OS architecture, and CPU byte-endianness. You cannot replicate between PostgreSQL 15 and PostgreSQL 16 using physical WAL shipping.

C. Logical (Row-Based) Replication / Change Data Capture (CDC)

The leader emits structured change records describing mutations at the row level (e.g. INSERT INTO users (id=1, name='Alice'), UPDATE balance SET val=50 WHERE id=1).

• Advantages: Engine-version agnostic. Enables zero-downtime database major upgrades, multi-master replication, and real-time streaming into Apache Kafka / Elasticsearch.

---

2. The Automated Failover Pipeline

When the primary database crashes or becomes unresponsive, the cluster must execute an automated Failover Protocol:

The 4 Stages of Safe Failover:

1. Failure Detection: Sentinel nodes exchange periodic heartbeat pings (e.g. every $1\text{s}$). If the leader does not respond within a configured lease timeout ($10\text{s}$), it is presumed dead.
2. Leader Election / Candidate Selection: The sentinel inspects all available replicas and elects the standby with the highest Log Sequence Number (LSN) to minimize or eliminate Recovery Point Objective (RPO) data loss.
3. Standby Promotion: The chosen replica exits standby recovery mode, applies any remaining uncommitted WAL buffers, and assumes the Leader role.
4. Traffic Reconfiguration: The routing layer (e.g. PgBouncer, HAProxy, AWS Route53 DNS, Kubernetes Service) is atomically updated to direct mutating write queries to the new leader.

---

3. The Split-Brain Catastrophe

The most dangerous failure mode in distributed databases is Split-Brain.

If a temporary network partition severs communication between the Sentinel and the Primary, the Sentinel will assume the Primary is dead and promote Standby B to become the new Leader.

However, if the Old Primary is still alive and reachable by a subset of application clients, both nodes will accept writes simultaneously:

Why Split-Brain Destroys Systems:

• Both nodes generate overlapping autoincrement primary keys.
• Account balances are mutated independently on both sides without mutual exclusion.
• When the network partition heals, the two WAL histories cannot be merged automatically. Manual data forensics and permanent data loss are unavoidable.

---

4. Fencing Tokens: The Definitive Protection Against Zombie Primaries

To mathematically prevent split-brain without relying on fragile network timing, distributed systems use Fencing Tokens (pioneered by Martin Kleppmann).

A fencing token is a strictly monotonically increasing integer (epoch number) generated by a consensus coordinator (ZooKeeper, etcd, Consul) every time a new leader is promoted.

Fencing Rule:

Every mutating storage write must include the leader’s active fencing token. The storage layer records the highest token it has ever observed (${\tau }_{\max }$).

$\text{If }{\tau }_{\text{incoming}}<{\tau }_{\max }⟹\text{REJECT WRITE IMMEDIATELY}$

Because Token 31 is strictly less than Token 32, the zombie primary’s write is safely rejected before it can corrupt storage.

---

5. Code Deep-Dive: Failover Orchestrator with Fencing Token Enforcement

---

6. Production Failure Postmortem: The GitHub 2018 Split-Brain Outage

Incident Overview:

In October 2018, GitHub experienced a 24-hour service degradation where database writes were locked and data had to be manually reconstructed after a brief 43-second network glitch between their East and West Coast data centers.

What Happened:

1. A brief network partition interrupted traffic between the primary US-East data center and the standby US-West data center.
2. The orchestrator detected the network blip and promoted the US-West replica to primary.
3. However, the original US-East primary was still active and accepting traffic from local US-East services.
4. For several minutes, both US-East and US-West wrote distinct, conflicting transactions to their MySQL databases.
5. When the partition reconnected, binary logs had diverged. To prevent silent data corruption, GitHub engineers had to halt all public writes and spend 24 hours manually reconciling diverged database rows.

Remediation:

• Deployed Raft-based consensus orchestration (Orchestrator + Consul) to mandate quorum consent before any node can be promoted.
• Enforced strict STONITH ("Shoot The Other Node In The Head") fencing protocols to power down or cut network interfaces on demoted primaries before standby promotion completes.

---

7. Chapter Summary & Failover Hardening Checklist