Distributed Locking with Redis: Redlock, Fencing & Failures

Distributed Locking with Redis: Redlock, Fencing & Failures

A Systems Architecture Deep Dive into Redlock, Martin Kleppmann's Critique, Fencing Tokens, and Concurrency Hazards

A distributed lock is a mutual exclusion primitive used by independent compute processes running on separate nodes in a network to coordinate shared access to a constrained resource. In modern distributed systems, Redis is widely chosen as the distributed lock manager (DLM) due to its sub-millisecond in-memory throughput, atomic key-expiration mechanics, and native Lua scripting execution.

However, distributed locking across an asynchronous network is one of the most deceptively complex problems in software architecture. A naive Redis lock implementation can lead to silent race conditions, split-brain lock acquisition, and catastrophic database corruption when subjected to network partitions, asynchronous replication failovers, or Stop-the-World garbage collection pauses.

---

---

1. The Core Tension: Efficiency Locks vs. Correctness Locks

Before evaluating algorithms or writing synchronization code, systems architects must establish the fundamental purpose of the lock. In distributed systems engineering, distributed locks fall into two completely distinct operational categories: Efficiency Locks and Correctness Locks.

Type 1: Efficiency (Optimization) Locks

The primary objective of an efficiency lock is resource optimization—preventing multiple worker processes from simultaneously performing the same expensive, idempotent computational task.

• Characteristics:
  • If the distributed lock occasionally fails (e.g., two worker nodes acquire the lock concurrently for 50 milliseconds due to a failover), the only consequence is minor wasted computation (such as processing an image twice or dispatching two identical marketing emails).
  • The system guarantees eventual consistency through idempotency keys.
  • Single-instance Redis locks or simple TTL-based distributed caches are ideal and cost-effective for efficiency locking.

---

Type 2: Correctness (Safety) Locks

The primary objective of a correctness lock is strict mutual exclusion and data integrity—ensuring that under no circumstances can two processes concurrently mutate shared state.

• Characteristics:
  • If mutual exclusion fails for even a few milliseconds, the database suffers irreversible corruption, split-brain writes, or double-spending financial transactions.
  • The lock is the sole barrier protecting invariant constraints across non-transactional downstream services.
  • Critical Insight: As proven by distributed systems researchers (Martin Kleppmann, Leslie Lamport), standard Redis locks without cryptographic fencing tokens are fundamentally unsafe for correctness locks over asynchronous networks.

---

2. In-Process Locking vs. Distributed Locking

Understanding why distributed locking is difficult requires comparing single-node operating system primitives with distributed network primitives.

---

3. Single-Instance Redis Lock: The Canonical Pattern

The standard industry implementation of a distributed lock on a single Redis instance relies on atomic key creation with an automatic expiration time (Time-To-Live, or TTL).

The Atomic Acquisition Command

To acquire a lock, the client sends the following command to Redis:

Where:

• lock:resource_id: The string key representing the locked resource.
• <unique_client_token>: A cryptographically random, unique identifier generated per acquisition attempt (e.g., UUIDv4, 128-bit random bytes, or nanoid).
• NX: Conditional flag meaning "Only set the key if it does NOT already exist" (provides mutual exclusion).
• PX 30000: Sets the key's automatic expiration to $30,000\text{ milliseconds}$ ($30\text{ seconds}$). This provides deadlock protection if the client crashes while holding the lock.

---

The Historical SETNX + EXPIRE Anti-Pattern

In early versions of Redis (prior to version 2.6.12), the SET command did not support the NX and PX parameters simultaneously. Developers frequently implemented locking using two sequential commands:

[!CAUTION] Why Multi-Command Locking is Fatal: If the client process crashes, loses power, encounters an Out-Of-Memory (OOM) kill, or loses network connectivity immediately after step 1, the EXPIRE command is never dispatched. The lock key remains in Redis indefinitely with no TTL, permanently deadlocking all other worker processes across the entire infrastructure until manual administrative intervention.

---

4. Atomic Release via Lua Script

Releasing a distributed lock is just as critical and error-prone as acquiring it. A naive developer might attempt to release the lock by sending a simple DEL lock:resource_id command.

The Lock Hijacking Race Condition

Consider what occurs when a simple DEL is executed in a production system subject to execution delays:

If Client 1's execution exceeds the TTL, its lock expires automatically. When Client 2 subsequently acquires the lock, Client 1 finally finishes and issues DEL lock:resource_id—inadvertently destroying Client 2's valid active lock and allowing Client 3 to acquire the mutex simultaneously!

---

The Canonical Release Lua Script

To prevent accidental lock deletion, the release operation must verify that the value stored in the key exactly matches the unique token generated by the releasing client before deleting it.

Because Redis executes Lua scripts atomically within its single-threaded event loop, checking the token and deleting the key are guaranteed to execute without interleaving commands:

Complete Production Implementation in TypeScript

---

5. The Lock Renewal / Watchdog Pattern

In real-world applications, estimating the exact maximum runtime of a business process is notoriously difficult. If the TTL is set too short, the lock may expire prematurely; if set too long, a crashed worker holds the lock for an excessive period, blocking other consumers.

Modern distributed lock clients (such as Redisson in Java or custom implementations in Go/Node.js) solve this through the Watchdog (Lock Renewal) Pattern.

The Watchdog Renewal Lua Script

To safely renew an active lock without race conditions, the background watchdog executes a renewal Lua script:

The Automatic Crash Safety Guarantee

If the worker node holding the lock suddenly crashes, suffers an unhandled exception, or loses network connectivity:

1. The background watchdog thread dies alongside the host process.
2. The watchdog stops sending renewal heartbeats.
3. Within the remaining TTL (at most $30\text{ seconds}$), Redis automatically expires and removes the key.
4. Other worker nodes resume normal lock acquisition without manual recovery!

---

6. The Master-Replica Asynchronous Replication Trap

In enterprise production deployments, running a single standalone Redis instance represents a catastrophic Single Point of Failure (SPOF). Consequently, engineering teams almost universally deploy Redis in a High Availability (HA) topology using Master-Replica replication with Redis Sentinel or Redis Cluster.

However, introducing replication to solve availability creates a lethal concurrency hazard that breaks distributed lock mutual exclusion.

---

The Anatomy of the Replication Race Condition

The root cause of this failure mode lies in Redis’s foundational architectural design: Redis replication is inherently asynchronous.

1. Client Write Acknowledgment: When Client 1 sends SET lock:resource token NX PX 30000, the Redis Master writes the key to its in-memory key-space and immediately returns +OK to Client 1.
2. Asynchronous Buffer Streaming: The write command is appended to the master's replication backlog buffer (repl-backlog). The master streams this buffer over a non-blocking TCP socket to connected replicas asynchronously in the background.
3. The Unreplicated Crash Window: There exists a non-zero time window ($\Delta t_{\text{rep}}\approx 0.5\text{ms}-50\text{ms}$, depending on network congestion and TCP socket buffers) where the lock key exists on the master but has not yet been received or executed by the replica.
4. Failover & Mutual Exclusion Collapse: If the master crashes (hardware failure, kernel panic, hypervisor eviction, or OOM killer) during $\Delta t_{\text{rep}}$:
   • Redis Sentinel or Redis Cluster detects the master's absence and promotes a replica to become the new active master.
   • The promoted replica has zero knowledge of the lock.
   • Client 2 requests the same lock, which is granted immediately.
   • Mutual exclusion is completely destroyed. Both Client 1 and Client 2 enter the critical section simultaneously, causing data corruption, double processing, or inconsistent state transitions.

---

7. Why Redis Sentinel & Redis Cluster Cannot Solve This

A common misconception among systems engineers is that upgrading from a single Redis instance to Redis Sentinel or a Redis Cluster solves the failover locking race condition.

In reality, neither Sentinel nor Cluster provides linearizable consistency.

CAP Theorem Classification: Redis is an AP System

Under Eric Brewer's CAP Theorem, database systems must choose between Consistency (Linearizability) and Availability during a Network Partition ($P$).

• CP Consensus Engines (Etcd / ZooKeeper): A write is never acknowledged to the client until a quorum (majority) of nodes has persisted and fsynced the transaction to their logs. If a partition occurs, minority partitions reject writes to preserve linearizable safety.
• AP In-Memory Engines (Redis): Designed for extreme low-latency caching and throughput. Redis favors availability and speed over linearizability, accepting writes on masters without waiting for replica confirmation.

---

The WAIT Command Illusion

To address asynchronous replication loss, Redis introduced the WAIT command:

WAIT numreplicas timeout blocks the client until the previous write command has been successfully transferred and acknowledged by at least numreplicas replicas, or until timeout milliseconds elapse.

Why WAIT Still Fails as a Distributed Lock Primitive:

1. Non-Atomic Composition: SET and WAIT are two distinct commands. If WAIT returns 0 (indicating a timeout before replicas acknowledged), the lock key remains set on the master. The client cannot determine whether the write reached replicas or not, leaving the system in an indeterminate state.
2. Lack of Consensus Rollback: If the master acknowledges WAIT to a replica on an isolated network partition, Sentinel can still declare the master dead and promote an un-updated replica on the primary partition. Redis does not roll back or abort writes during failovers.
3. Severe Latency Penalty: Executing WAIT forces Redis to incur synchronous network round-trips to replicas, destroying the single greatest benefit of Redis: sub-millisecond execution speed.

---

8. Network Partitions & Split-Brain Scenarios in Redis Cluster

In a multi-master Redis Cluster, key slots are partitioned across master nodes. During a network partition, a cluster split-brain can allow multiple masters to accept conflicting writes for the exact same hash slot.

The cluster-node-timeout Dual-Master Window

When a Redis Cluster master becomes isolated on a minority partition:

1. The isolated master continues accepting read and write commands from clients connected to its partition for up to cluster-node-timeout (typically configured between $5\text{ to }15\text{ seconds}$).
2. Simultaneously, the majority partition nodes observe that the master is unresponsive, wait for cluster-node-timeout, and elect the replica to become the new master for that slot.
3. During this multi-second window, two distinct master processes accept conflicting lock commands for the same resource simultaneously.
4. When the network partition heals, the old master is demoted to a replica and executes a full sync (PSYNC), silently discarding all writes made to it during the partition—including the lock acquisition that Client 1 believed was valid!

---

9. Salvatore Sanfilippo's Solution: The Redlock Algorithm

To overcome the Single Point of Failure (SPOF) of standalone instances and the replication race conditions of master-replica pairs without deploying a heavyweight Paxos or Raft cluster, Salvatore Sanfilippo (Antirez, the creator of Redis) proposed the Redlock Algorithm.

Redlock replaces the single-master model with a multi-master quorum consensus protocol.

---

Architectural Foundations of Redlock

1. Independent Master Nodes ($N=5$): The architecture mandates $N$ completely independent Redis master instances running on separate physical hardware servers or availability zones.
   • No Asynchronous Replication: There are no replica nodes.
   • No Cluster Coordination: The nodes do not communicate with each other (no gossip protocol, no Raft heartbeats, no hash slots).
   • Pure Client-Side Orchestration: All quorum negotiation, timing calculations, and rollback logic are driven entirely by the client library.
2. Quorum Fault Tolerance ($M=\lfloor N/2\rfloor +1$): In a 5-node setup, the required quorum is $M=3$ nodes. The system can tolerate up to $F=2$ dead or partitioned nodes without losing availability or safety.

---

10. The 5-Step Redlock Acquisition Protocol

To acquire a distributed lock on a resource across $N$ instances with a total lease time of $\text{TTL}$, a client executes the following protocol:

---

Step 1: High-Resolution Monotonic Timestamp ($T_1$)

The client captures the current monotonic timestamp $T_1$ with millisecond or microsecond precision using the operating system's monotonic clock (e.g., clock_gettime(CLOCK_MONOTONIC) in C/Linux or process.hrtime() in Node.js).

[!IMPORTANT] Why Wall-Clock Time (System.currentTimeMillis()) is Prohibited: Wall clocks are subject to Network Time Protocol (NTP) adjustments, leap seconds, and manual clock skew, which can make time appear to jump backwards or forwards. Monotonic clocks guarantee strictly increasing time deltas.

---

Step 2: Sequential Node Acquisition with Short Timeouts

The client attempts to acquire the lock on all $N$ instances sequentially (or in parallel) using the identical key name (lock:resource_id) and random token value (<client_token>).

To prevent the client from blocking if a node is partitioned or crashed, each network socket operation uses a connection and read timeout that is very small compared to the total lock TTL:

$\text{Socket Timeout}\ll \text{TTL}(\text{e.g., Socket Timeout}=5\text{ms}-50\text{ms}\text{ for a }10,000\text{ms TTL})$

If an instance fails to respond within $50\text{ms}$, the client immediately flags that node as failed and advances to the next instance.

---

Step 3: Quorum Check & Time Elapsed Calculation

The client measures the timestamp $T_2$ immediately after attempting all $N$ instances and calculates the total round-trip elapsed time:

$\Delta T=T_2-T_1$

The client checks two mandatory conditions:

1. Majority Quorum Condition: The lock was successfully acquired on at least $M=\lfloor N/2\rfloor +1$ instances (e.g., $\ge 3$ out of 5).
2. Validity Time Condition: The remaining lock validity time is strictly positive: $\text{Validity Time}=\text{TTL}-\Delta T-\text{Clock Drift}$

Where Clock Drift accounts for physical crystal oscillator drift across independent servers:

$\text{Clock Drift}=(\text{TTL}\times \text{DriftFactor})+2\text{ms}(\text{where DriftFactor}\approx 0.002=0.2\%)$

---

Step 4: Critical Section Execution

• Success Case: If both quorum is achieved ($M\ge 3$) and $\text{Validity Time}>0$, the lock is successfully acquired. The client holds the lock exclusively for the duration of $\text{Validity Time}$.
• Failure / Abort Case: If the client failed to acquire a majority (e.g., only acquired 2 nodes out of 5) OR the elapsed time $\Delta T$ exceeded the TTL ($\text{Validity Time}\le 0$), the lock acquisition is considered failed.

---

Step 5: The Mandatory Unlock-All Rollback

If lock acquisition fails for any reason, the client must dispatch the release Lua script to ALL $N$ instances—including the instances that returned socket errors or failed to respond.

[!WARNING] Why You Must Unlock Failed Nodes: If Node 3 processed the SET command in its memory buffer but the network connection dropped before sending the +OK packet back to the client, the client believes Node 3 failed. If the client does not send an unlock command to Node 3, that key will remain locked on Node 3 until its TTL expires, unnecessarily reducing quorum availability for subsequent lock attempts.

---

11. Production Redlock Implementation & Concurrency Traps

Production Redlock Engine in Go

---

The Split-Vote Livelock Hazard

When multiple worker nodes concurrently attempt to lock the same resource across an uncoordinated 5-node Redlock cluster, a split-vote livelock can occur:

• Client 1 locks Node 1 and Node 2.
• Client 2 locks Node 3 and Node 4.
• Client 3 locks Node 5.
• No single client achieves the 3-node quorum. All three clients fail, release their nodes, and immediately retry—potentially causing repeated collision livelocks.

Production Mitigation: Randomized Exponential Jitter

When a client fails to acquire a Redlock quorum, it must not retry immediately. It must sleep for a random backoff interval before retrying:

$\text{Backoff Interval}=\text{base\_delay}\times 2^{\text{attempt}}+\text{uniform}(0,\text{jitter\_window})$

Randomized jitter desynchronizes the retry loops of concurrent clients, allowing one client to claim the majority quorum on the subsequent attempt.

---

The Node Restart Hazard & The Delayed Restart Rule

Consider what occurs if one of the 5 Redlock master nodes crashes while holding active lock keys in memory:

If Redis persistence (AOF with fsync=always) is disabled or configured with fsync=everysec (the standard production default), Node 3 loses writes that occurred within the last second upon crashing. If Node 3 restarts immediately, it accepts Client 2's lock request, allowing Client 2 to form a 3-node quorum on Nodes 3, 4, and 5 while Client 1 is actively executing on Nodes 1, 2, and 3.

The Delayed Restart Rule:

To preserve Redlock safety without paying the catastrophic performance penalty of synchronous fsync=always disk writes on every command:

• When a Redlock master instance crashes, its system daemon (e.g., systemd / Kubernetes probe) must delay restarting the process for at least ${\text{TTL}}_{\max }$.
• By waiting longer than the maximum possible lock TTL, all existing lock keys that were held on the crashed node are guaranteed to have expired on the remaining running nodes before the restarted node rejoins the quorum.

---

12. The Martin Kleppmann Critique: The Asynchronous System Model

In February 2016, distributed systems researcher Martin Kleppmann (University of Cambridge, author of Designing Data-Intensive Applications) published a landmark analysis titled "How to do distributed locking".

Kleppmann proved that Redlock is fundamentally unsafe for correctness locking because its safety relies on assumptions of physical time and bounded network latency that do not hold in real-world asynchronous distributed systems.

---

The Three Asynchronous Hazards

In theoretical distributed computing, production networks (AWS, GCP, bare metal datacenters) operate under the Asynchronous System Model:

• Nodes have no shared memory.
• Message delivery delays are unbounded (a network packet may arrive in $1\text{ms}$ or $45\text{ seconds}$).
• Process execution speeds are arbitrary (a thread can pause at any instruction for an arbitrary duration).
• Hardware clocks are unreliable and cannot be trusted for safety invariants.

Under this model, algorithms that rely on elapsed wall-clock time to enforce mutual exclusion inevitably fail.

---

13. The Process Pause Race Condition: The "Zombie Writer"

The most devastating failure mode demonstrated by Kleppmann is the Zombie Writer Phenomenon caused by process pauses.

---

Step-by-Step Breakdown of the Catastrophe:

1. Lock Grant: Client 1 requests a distributed lock on account_42 with a $\text{TTL}=10\text{ seconds}$. Redlock grants the lock across 5 nodes.
2. The Unannounced Pause: Immediately after receiving the lock, Client 1’s runtime enters an unannounced execution pause:
   • JVM / Go Stop-the-World GC: Full Garbage Collection pauses on large heaps ($>32\text{GB}$) frequently take several seconds.
   • Memory Swapping / Page Faults: If the OS runs out of RAM, reading a memory page from swap disk can block a thread for hundreds of milliseconds.
   • Hypervisor CPU Steal Time: In virtualized cloud environments (e.g., AWS EC2 burstable instances), the hypervisor can suspend a guest VM to serve other tenants.
3. Silent Lock Expiry: While Client 1 is completely frozen, physical time continues to advance. The $10\text{ seconds}$ TTL elapses, and the lock keys expire across the Redlock cluster.
4. Legitimate Lock Re-grant: Client 2 attempts to lock account_42. Because the keys have expired, Redlock successfully grants the lock to Client 2.
5. Client 2 Mutates State: Client 2 reads the current balance (\100$),addsa$$50$credit,writesthenewbalance($$150$) to the database, and commits.
6. The Zombie Awakes: Client 1 resumes execution. A process has no native ability to sense that physical time has passed while it was asleep. Client 1 executes the instruction immediately following lock acquisition, believing it is the sole owner of the mutex.
7. Silent Data Corruption: Client 1 writes its stale computation (\100 - $30 = $70$) directly over Client 2’s write. Client 2’s transaction is permanently lost, leaving the database corrupted and the financial balance desynchronized.

[!CAUTION] Why Watchdog Heartbeats Cannot Prevent This: A common developer assumption is that a background watchdog thread would extend the TTL and prevent expiry. However, during a JVM Stop-the-World GC pause or OS hypervisor freeze, the entire operating system process is frozen—including all background threads and watchdogs. The watchdog is rendered completely powerless.

---

14. The Universal Solution: Monotonically Increasing Fencing Tokens

Kleppmann established that in an asynchronous distributed system, a lock manager cannot guarantee mutual exclusion on its own.

To achieve correctness, the storage layer must actively participate in enforcing mutual exclusion using Fencing Tokens.

---

The Fencing Token Protocol Specification

A fencing token is a strictly monotonically increasing number (generated via an atomic counter or consensus log revision) issued alongside every granted lock.

The Three Invariant Rules of Fencing:

1. Monotonic Generation: Every time the distributed lock manager grants a lock (even for the same resource to a different client), it increments a global 64-bit integer counter: ${\text{Token}}_{k+1}>{\text{Token}}_k(\text{e.g., }33\to 34\to 35)$
2. Token Propagation: The client must include this fencing token in every remote RPC, database query, and storage mutation executed within the critical section.
3. Storage-Side Gatekeeping: The storage system maintains the highest fencing token it has ever observed (last_fencing_token). If a client attempts a write with a token that is less than or equal to last_fencing_token, the storage layer unconditionally rejects the write.

---

15. Implementing Fencing Tokens Across Storage Engines

1. Relational SQL Databases (PostgreSQL / MySQL)

In an ACID relational database, fencing tokens are enforced via atomic conditional updates:

Application Handling in Node.js / Go:

---

2. NoSQL / Document Stores (Amazon DynamoDB)

In DynamoDB, fencing tokens are enforced using Condition Expressions:

---

3. Object Storage (AWS S3 / Google Cloud Storage)

For object stores that lack relational update statements, fencing is implemented using ETags (Optimistic Concurrency Control) or Conditional Put Object Headers (If-Match / If-None-Match):

---

Can Redis Generate Valid Fencing Tokens?

A crucial realization in the Kleppmann vs. Antirez debate is: Can Redis issue monotonically increasing fencing tokens?

• In a single-instance Redis, the server can increment an atomic 64-bit integer (INCR lock:counter:resource_id) alongside the lock grant.
• In a multi-master Redlock cluster, instances are completely uncoordinated and independent. There is no shared monotonic counter across the 5 nodes without running a consensus protocol (like Raft).
• The Irony of Redlock: If you already have a consensus system capable of generating linearizable monotonic tokens, you already have a CP consensus system and do not need Redlock!

---

16. Strong Consensus Alternatives: etcd, ZooKeeper & Postgres Locks

When an application requires Correctness Locking (Safety Locks) where duplicate execution causes financial losses, state desynchronization, or data corruption, systems architects should bypass Redis and deploy a dedicated CP Consensus Engine or an In-Database ACID Mutex.

---

1. etcd (Raft Protocol & Leases)

etcd is a strongly consistent, distributed key-value store built on the Raft consensus algorithm. It is the primary state store behind Kubernetes.

How etcd Distributed Locking Works:

1. TTL Leases with Heartbeats: The client creates a lease with a specified TTL (e.g., $10\text{ seconds}$) and streams periodic keep-alive heartbeats to the leader.
2. Atomic Transactional Compare-And-Swap (CAS): The client submits an atomic transaction:
   text

   1 lineCopy

   Loading code editor...
3. Built-in Monotonic Fencing Tokens: Every mutation committed to etcd increments a global 64-bit counter called the ModRevision. When a lock is granted, etcd returns this revision number, providing a natively linearizable fencing token with zero extra infrastructure!

---

2. Apache ZooKeeper (ZAB Protocol & Ephemeral Sequential ZNodes)

Apache ZooKeeper is a battle-tested coordination service used by Kafka, Hadoop, and HBase.

The Ephemeral Sequential Node Lock Recipe:

1. When a client wants to lock /locks/order_42, it creates an Ephemeral Sequential ZNode:
   text

   3 linesCopy

   Loading code editor...
2. The client fetches all children of /locks/order_42.
3. Lock Grant Condition: If the client's node has the lowest sequence number, the client holds the lock!
4. The Thundering Herd Elimination: If the client does not hold the lowest sequence number, it places a watcher strictly on the immediately preceding znode (node_i - 1). It does not poll. When the preceding client finishes and deletes its node, ZooKeeper fires an asynchronous event waking up only the next waiting client in the queue!
5. Crash Safety: Because the znodes are ephemeral, if the client holding the lock crashes or experiences a network partition, ZooKeeper’s session timeout automatically deletes the znode, cleanly passing the lock to the next worker.

---

3. PostgreSQL Advisory Locks

If your application already relies on PostgreSQL as its primary transactional database, you can achieve ACID-compliant distributed locking without deploying Redis, etcd, or ZooKeeper.

PostgreSQL provides application-level mutexes called Advisory Locks:

Why Postgres Advisory Locks are Superior for ACID Workloads:

• Zero Infrastructure Sprawl: Uses your existing highly-available, backed-up PostgreSQL cluster.
• Transaction Lifecycle Binding: pg_advisory_xact_lock is strictly bound to the database transaction. It is physically impossible for a client to forget to release the lock or for the lock to expire prematurely while the transaction is still running.
• Crash Proof: If the application worker dies, the TCP connection to PostgreSQL drops, and the database kernel immediately releases the advisory lock.

---

17. Architectural Decision & Trade-off Matrix

---

18. Frequently Asked Questions (GEO & Search Engine Optimized)

What does distributed locking mean?

Distributed locking is a synchronization mechanism used in computer networks to ensure that multiple independent worker processes or microservices do not concurrently execute the same critical section or mutate the same shared resource. It extends single-process mutex concepts across network and container boundaries using shared coordination services like Redis, etcd, ZooKeeper, or PostgreSQL.

What is the difference between Redis distributed locks and ZooKeeper locks?

Redis distributed locks rely on key expiration (TTL) and in-memory key-value operations (SET NX PX). Because Redis uses asynchronous replication and relies on physical time, it is optimized for high-throughput efficiency locking. In contrast, Apache ZooKeeper uses the ZAB consensus protocol with Ephemeral Sequential znodes and event-driven watchers. ZooKeeper guarantees linearizability (CP in CAP theorem), making it safe for strict correctness locking where data corruption must be prevented at all costs.

Is Redlock safe for financial transactions?

No. Distributed systems research (most notably by Martin Kleppmann) has proven that Redlock is unsafe for financial transactions or state mutations requiring strict correctness. Redlock relies on physical time bounds that are invalidated by Stop-the-World garbage collection pauses, network latency spikes, and hardware clock drift. For financial transactions, engineers should use CP consensus engines with fencing tokens (such as etcd or ZooKeeper) or database-native ACID transactions with row-level or advisory locks.

Is Redis a single point of failure?

A standalone single-instance Redis deployment is a Single Point of Failure (SPOF). However, if you add Master-Replica replication with Sentinel or Cluster, failovers become asynchronous, introducing a race condition where the newly promoted master does not have the lock key. Redlock attempts to solve this SPOF by using 5 independent master nodes, but introduces operational complexity and timing vulnerabilities.

How do fencing tokens prevent race conditions in distributed systems?

A fencing token is a strictly monotonically increasing sequence number granted alongside a distributed lock. When a worker process writes to a database or storage service, it includes this token. The storage layer records the highest token it has processed and unconditionally rejects any write with an older or duplicate token. This prevents a "zombie" worker that was paused by a garbage collection cycle from overwriting newer, valid data.

---

19. Architectural Decision Tree & Interactive Arena Challenge

When choosing a distributed locking strategy for your architecture, follow this staff-engineer decision framework:

---

🎮 Ready to Master Distributed Systems & Consensus?

Test your understanding of distributed mutexes, consensus algorithms, replication lag, and concurrency hazards with interactive hands-on challenges on the Initnode Arena:

👉 Explore Interactive Distributed Systems & AI Challenges on Initnode Arena