Multi-Region Active-Passive vs Active-Active Deployments

In mission-critical enterprise systems, relying on a single cloud region (e.g. us-east-1) leaves your application vulnerable to catastrophic datacenter fires, submarine fiber optic cuts, cooling failures, and regional cloud provider outages.

To achieve 99.999% availability (Five Nines), modern systems deploy across multiple geographic regions worldwide.

Designing a multi-region architecture requires balancing the fundamental trade-offs between Recovery Objectives (RPO vs RTO), Network Latency (Speed of Light), and Data Consistency.

---

1. Defining Disaster Recovery Objectives: RPO and RTO

Every multi-region strategy is defined by two fundamental business metrics:

---

2. Active-Passive Deployments: Cold vs Warm vs Hot Standby

In an Active-Passive architecture, all write traffic routes to a single Primary (Active) Region. A secondary region sits in reserve awaiting a disaster failover.

The Synchronous vs Asynchronous Cross-Region Dilemma:

• Synchronous Replication (RPO = 0): Every write must wait for an acknowledgment across transatlantic fiber ($70-100\text{ms}$). Write throughput drops by $90\%$.
• Asynchronous Replication (Low Latency): Writes commit locally in $<2\text{ms}$. However, if the primary region vaporizes, any transactions in transit across the WAN are permanently lost ($\text{RPO}>0$).

---

3. Active-Active Deployments (Multi-Region Writes)

In an Active-Active architecture, microservices and databases in all regions accept live write traffic simultaneously.

The 3 Core Architectural Approaches for Active-Active Writes:

1. Partitioned / Sharded Locality (Pinning):
   • European users write to EU shards; American users write to US shards.
   • Cross-region writes are eliminated for $99\%$ of traffic.
2. Conflict-Free Replicated Data Types (CRDTs):
   • Used by Amazon DynamoDB Global Tables, Redis Enterprise, and Riak.
   • Concurrent writes merge mathematically without data loss.
3. Globally Synchronous Consensus (TrueTime / Spanner):
   • Uses Paxos consensus groups spanning multiple regions with TrueTime commit wait.

---

4. Global Ingress: Anycast BGP vs Geo-DNS Routing

How do you route millions of global users to the nearest operational region?

---

5. Production Failure Postmortem: The DNS TTL Failover Outage

Incident Overview:

In 2021, a global SaaS provider experienced an 80-minute outage when us-east-1 suffered a power outage, despite having a fully provisioned warm standby in us-west-2.

What Happened:

1. When us-east-1 crashed, the automated health-check daemon updated the Geo-DNS Route 53 records to point api.company.com to us-west-2.
2. However, the operations team had configured the DNS A record with a TTL = 86400 ($24\text{ hours}$).
3. Over $75\%$ of client ISP resolvers, mobile apps, and third-party API clients cached the dead IP address.
4. Despite us-west-2 being 100% operational, incoming traffic hammered the dead us-east-1 load balancer for over an hour.

Remediation:

• Set all public DNS entry points to TTL = 60 seconds.
• Migrated primary public ingress to Anycast BGP (AWS Global Accelerator) with automated edge health-checking, reducing failover convergence from hours to $<10\text{ seconds}$.

---

6. Chapter Summary & Multi-Region Invariants