Distributed Locking Mechanisms

In a single-process server, synchronizing access to shared resources is handled by kernel mutexes and language primitives (e.g. pthread_mutex, Go sync.Mutex, Java synchronized).

In a distributed system, processes run on independent machines without shared memory, communication is asynchronous, and servers experience unpredictable network delays, clock drift, and Stop-the-World Garbage Collection (GC) pauses.

Implementing a correct distributed lock requires understanding why simple TTL locks fail and how to use Fencing Tokens and Consensus-backed Leases.

---

1. The Fatal Flaw of Simple TTL Locks

Many developers implement distributed locks using simple Redis keys with TTLs:

The Martin Kleppmann Breakdown (2016):

In his famous analysis "How to do distributed locking", Martin Kleppmann proved that a simple TTL lock cannot guarantee mutual exclusion:

---

2. The Solution: Fencing Tokens

To guarantee safety in an asynchronous network, a lock service cannot rely on client timers alone. The target Storage Layer MUST actively validate incoming writes using a Fencing Token.

The Fencing Invariant:

1. Every time a lock is acquired, the lock service issues a monotonically increasing integer (Fencing Token): $33,34,35,\dots$
2. The client attaches its fencing token to every read and write request sent to the storage service.
3. The storage service tracks the highest fencing token it has ever observed:

$\text{If }{\text{Token}}_{\text{incoming}}<{\text{Token}}_{\text{max\_observed}}⟹\text{REJECT WRITE}$

---

3. Redis Redlock vs etcd / ZooKeeper Leases

How should you choose a distributed locking infrastructure?

Why etcd / ZooKeeper are Superior for Critical Data:

• Consensus-backed: etcd uses Raft consensus, ensuring that a lock is replicated to a majority quorum before being acknowledged.
• Dynamic Heartbeat Leases: The client sends periodic keep-alive heartbeats. If the client dies or undergoes a prolonged pause, etcd revokes the lease automatically without waiting for arbitrary fixed TTLs.
• Native Fencing Revisions: etcd generates monotonic 64-bit revision numbers that serve directly as perfect fencing tokens.

---

4. Code Deep-Dive: Storage-Level Fencing Token Validator

---

5. Production Failure Postmortem: The Unfenced S3 Overwrite Corruption

Incident Overview:

In 2020, a cloud analytics platform experienced intermittent data corruption where older daily report aggregates silently overwrote newly computed customer analytics on AWS S3.

What Happened:

1. Worker A acquired a Redis lock with a $30\text{-second}$ TTL and started aggregating 50 million database rows.
2. An unexpected network socket timeout on the database query caused Worker A to stall for $45\text{ seconds}$.
3. The Redis lock TTL expired, and the lock scheduler assigned Worker B to run the same job.
4. Worker B completed its run and uploaded analytics_2020_11_01.parquet (Version 2) to S3.
5. Worker A woke up, assumed it was still the sole owner of the job, and executed its s3.putObject() call, overwriting Version 2 with its stale, incomplete Version 1 file.

Key Lesson:

• Distributed locks cannot protect passive storage that lacks fencing or version checking. Always use conditional updates (If-Match: ETag in S3, or monotonically increasing fencing tokens in databases) so the storage layer rejects stale worker writes.

---

6. Chapter Summary & Distributed Locking Rules