Single-Leader Replication: Synchronous vs Asynchronous

In a distributed database, a single physical server cannot withstand hardware failures or scale to millions of concurrent read queries. To achieve fault tolerance, high availability, and horizontal read scaling, data must be copied across multiple independent machines—a process known as replication.

The foundational replication topology used in enterprise relational and document databases (including PostgreSQL, MySQL, MongoDB, and Redis) is Single-Leader Replication (also called Primary-Backup or Master-Slave).

---

1. The Single-Leader Architecture

In a single-leader cluster, every node is assigned a specific role:

1. The Leader (Primary / Master):
   • All write requests (INSERT, UPDATE, DELETE, DDL) must be routed directly to the leader.
   • The leader validates the write, writes it sequentially to its Write-Ahead Log (WAL), and broadcasts the change stream to all replicas.
2. The Followers (Read Replicas / Standbys):
   • Followers accept read-only queries from application clients.
   • They consume the leader’s replication stream (physical WAL records, logical row events, or statement logs) and apply mutations locally in the exact sequential order they were committed on the leader.

---

2. Synchronous vs. Asynchronous Replication Trade-offs

The defining architectural decision in single-leader topologies is when the leader acknowledges the write to the client:

Architectural Comparison Matrix:

[!IMPORTANT] Why Fully Synchronous Clusters are Rare in Production:
If a database cluster has 5 fully synchronous followers, each with $99.9\%$ individual availability, the total write availability of the cluster drops to: $P(\text{Available})=0.999^5\approx 99.5\%$ Adding more synchronous read replicas actually decreases your system's write uptime. Therefore, production systems almost universally adopt Semi-Synchronous configuration (e.g. synchronous_commit = on with synchronous_standby_names = 'FIRST 1 (rep1, rep2)' in PostgreSQL).

---

3. Replication Lag Anomalies and Inconsistencies

When applications scale out by routing read queries to asynchronous followers, network delays or heavy follower query loads create Replication Lag ($\Delta t$).

This lag breaks naive application assumptions and introduces three classic consistency anomalies:

A. Reading Your Own Writes (Read-After-Write Inconsistency)

A user submits a comment on a social network. The write commits to the Leader. The browser immediately reloads the page, and the load balancer routes the read to a Follower that is $200\text{ms}$ behind. The user does not see their own comment and assumes the system failed.

B. Monotonic Reads (Moving Backward in Time)

A user refreshes their inbox twice. The first request hits Follower 1 (lag = $5\text{ms}$), showing an unread email. The second request hits Follower 2 (lag = $1500\text{ms}$). The unread email suddenly disappears. Time appears to move backward.

C. Consistent Prefix Reads (Violation of Causality)

If Write B is caused by Write A (e.g. Question $\to$ Answer), an async replica that applies transactions out of causal order might display the Answer before the Question, confusing users.

---

4. Architectural Solutions for Read-Your-Own-Writes Consistency

To eliminate replication lag anomalies without sacrificing read scaling, distributed architects use three proven patterns:

1. User Profile Pinning: Always read resources that the user can edit (such as user profile, settings, account balance) from the Leader, while reading public feeds from Followers.
2. Timestamp / LSN Caching: When a user writes data, return the current Log Sequence Number (LSN) or timestamp in a response cookie (last_write_lsn = 18492048). On subsequent reads, only query followers whose pg_last_wal_replay_lsn() is $\ge \text{last\_write\_lsn}$.
3. Write Window Tracking: After any mutating request, pin all reads for that specific user session to the Leader for a safety window (e.g. $5\text{ seconds}$), after which traffic reverts to followers.

---

5. Code Deep-Dive: Read-Write Splitting Router with Sticky Session Pinning

---

6. Production Failure Postmortem: The Lost Orders Failover Disaster

The Outage:

A retail logistics company suffered a sudden hardware power failure on their PostgreSQL primary server. The automated failover orchestrator immediately promoted an asynchronous read replica to become the new primary.

What Went Wrong:

1. The cluster was configured with pure Asynchronous Replication (synchronous_commit = off).
2. At the moment of the crash, the primary had committed $450$ customer purchase orders in local memory that had not yet been transferred over the network to the replica (replication lag was $\approx 800\text{ms}$).
3. When the replica was promoted to Leader, those $450$ transactions simply did not exist on the new leader.
4. Even worse, when the old primary was restarted, its WAL history had diverged (Split-Brain WAL Conflict), forcing engineers to manually reconcile $450$ missing credit card charges from Stripe webhook logs.

Remediation:

• Transitioned cluster to Semi-Synchronous Replication (synchronous_commit = on with synchronous_standby_names = 'ANY 1 (replica_az1, replica_az2)').
• Writes to critical ledger tables are guaranteed to be fsynced on at least two independent Availability Zones before acknowledging success to the client.

---

7. Chapter Summary & Configuration Cheatsheet