The PACELC Theorem: Consistency vs Latency in Normal Operations

While the CAP Theorem explains how distributed databases behave during rare network partitions ($P$), partitions occur during less than $0.01\%$ of total cluster uptime.

What trade-offs do distributed databases make during the remaining 99.99% of normal, healthy operations?

In 2012, Yale Professor Daniel Abadi formulated the PACELC Theorem to capture the full spectrum of distributed database trade-offs:

$\text{If }\mathbf{P}\text{ (Partitioned): Choose }\mathbf{A}\text{ vs }\mathbf{C}|\mathbf{E}\text{lse (Normal Operation): Choose }\mathbf{L}\text{ (Latency) vs }\mathbf{C}\text{ (Consistency)}$

---

1. The Physics of the "E" Trade-off: Latency vs Consistency

Why does strong consistency inevitably cost latency during normal operations? The physical speed of light in optical fiber.

Light in silica glass travels at approximately $200,000\text{ km/s}$ ($\approx 5\text{ microseconds per kilometer}$).

The Inescapable Latency Formula:

1. If you choose Strong Consistency ($\mathbf{EC}$): Every write must wait for a synchronous network acknowledgment from remote cross-region replicas before responding 200 OK to the client. $\text{Write Latency}\ge {\text{RTT}}_{\text{cross-region}}\approx 70\text{ms}-120\text{ms}$
2. If you choose Low Latency ($\mathbf{EL}$): The leader commits the write to local SSD storage ($<2\text{ms}$) and acknowledges the client immediately, streaming updates to remote replicas asynchronously in the background. $\text{Write Latency}=\text{Local Disk Append}\approx 1\text{ms}-3\text{ms}$

---

2. The 4 PACELC Database Quadrants

Every distributed storage system in existence maps cleanly into one of four PACELC quadrants:

---

Detailed Breakdown of Each Quadrant:

1. $PA/EL$ (Available under Partition, Low Latency Normally)

• Examples: Apache Cassandra, Amazon DynamoDB (default), Riak.
• Philosophy: Extreme uptime and sub-millisecond response times at all costs.
• Trade-off: Replicas can return stale data or suffer write conflicts during both normal operations and network partitions.

2. $PC/EC$ (Consistent under Partition, Consistent Normally)

• Examples: Google Spanner, CockroachDB, etcd, Consul.
• Philosophy: Absolute linearizability and transactional integrity.
• Trade-off: High write latencies ($70\text{ms}-150\text{ms}$ cross-region) and refusal to serve requests if consensus quorum is lost.

3. $PC/EL$ (Consistent under Partition, Low Latency Normally)

• Examples: PostgreSQL / MySQL with single-leader asynchronous replication.
• Philosophy: During normal operations, the leader acknowledges writes immediately without waiting for replicas ($\mathbf{EL}$). If the leader crashes, the cluster fails over safely ($\mathbf{PC}$) without allowing dual-master writes.

4. $PA/EC$ (Available under Partition, Consistent Normally)

• Examples: MongoDB configured with w: "majority" writes and readPreference: "secondaryPreferred".
• Philosophy: Synchronously commits writes to a majority under normal conditions ($\mathbf{EC}$), but allows partitioned secondaries to continue serving stale reads ($\mathbf{PA}$).

---

3. Visual Blueprint: PACELC Latency-Consistency Boundary

---

4. Code Deep-Dive: A Configurable PACELC Storage Engine

---

5. Production Failure Postmortem: DynamoDB Global Tables Inventory Oversell

Incident Overview:

During a major Black Friday flash sale, a global retail platform used Amazon DynamoDB Global Tables (a $PA/EL$ system) across us-east-1 (Virginia) and eu-west-1 (Ireland) to manage a limited stock of 500 limited-edition gaming consoles.

What Happened:

1. The inventory was decremented concurrently by US and European shoppers.
2. Because DynamoDB Global Tables operate under the $\mathbf{EL}$ model (asynchronous replication), a purchase in Virginia updated the local US table in $2\text{ms}$, but took $150\text{ms}$ to replicate across the Atlantic.
3. Within that $150\text{ms}$ replication window, European shoppers read the local Dublin replica, which still showed stock remaining.
4. When the two cross-region streams merged using Last-Write-Wins ($\mathbf{PA}$), 500 items were sold 842 times, resulting in 342 backordered cancellations and massive customer backlash.

Remediation:

• Migrated shared finite inventory counters to a $PC/EC$ transactional database (CockroachDB / Amazon Aurora Global Database with Write Forwarding).
• Retained DynamoDB $PA/EL$ for non-blocking operations (shopping carts, user clickstream logs, product catalog views).

---

6. Chapter Summary & PACELC Database Selection Guide