Log Compaction, Snapshotting, and Membership Changes

In a long-running distributed system, an append-only consensus log cannot grow indefinitely.

A database processing $10,000\text{ writes/sec}$ accumulates nearly 1 billion log entries per day, rapidly exhausting disk and memory capacity and making node restarts prohibitively slow.

To remain bounded, production consensus engines (etcd, CockroachDB, Consul) use Log Compaction and State Machine Snapshotting.

---

1. How State Machine Snapshotting Works

In snapshotting, the entire current state of the deterministic state machine is written to permanent storage, and all preceding log entries are safely discarded:

Snapshot Metadata:

• lastIncludedIndex: The highest log index applied to the state machine at the time the snapshot was created.
• lastIncludedTerm: The term of that log entry.
• All entries up to lastIncludedIndex are purged from memory and disk.

---

2. Catching Up Lagging Nodes: The InstallSnapshot RPC

When a follower node is partitioned for days or a brand-new node joins the cluster, the leader may have already discarded the log entries needed to bring the node up to date via AppendEntries.

In this scenario, the leader streams a snapshot using the InstallSnapshot RPC:

---

3. Dynamic Cluster Membership Changes

How do you safely add or remove servers from a live consensus cluster (e.g. expanding from 3 nodes to 5 nodes) without taking downtime?

The Split-Brain Hazard of Naive Configuration Changes:

If each server switches from Old Configuration ($C_{\text{old}}$) to New Configuration ($C_{\text{new}}$) at its own independent local time, there is a window where two separate servers can form two conflicting majority quorums simultaneously:

---

Solution 1: Joint Consensus ($C_{\text{old},\text{new}}$)

In the full Raft specification, configuration changes transition through a transitional two-phase state called Joint Consensus:

1. The cluster transitions into configuration $C_{\text{old},\text{new}}$.
2. Any log entry or election requires separate majority agreement from BOTH $C_{\text{old}}$ AND $C_{\text{new}}$.
3. Once $C_{\text{old},\text{new}}$ is committed, the leader proposes $C_{\text{new}}$ as a normal log entry.

Solution 2: Single-Server Changes (Industry Standard)

A simpler alternative (used by etcd and HashiCorp Raft) is Single-Server Membership Change:

• The cluster is only permitted to add or remove one single server at a time ($3\to 4\to 5$).
• Mathematical proof: Adding or removing a single server makes it impossible for $C_{\text{old}}$ and $C_{\text{new}}$ majorities to overlap disjointly.

---

4. Code Deep-Dive: InstallSnapshot Handler

---

5. Production Failure Postmortem: The Snapshotting Disk I/O Starvation Outage

Incident Overview:

In 2020, a production etcd cluster hosting Kubernetes control-plane state experienced continuous cascading node crashes during scheduled cluster-wide snapshot compaction.

What Happened:

1. The etcd cluster accumulated 8GB of historical key versions.
2. The automatic snapshot compaction thread serialized the entire in-memory B-bolt database to disk synchronously on the main thread.
3. The intense disk I/O saturated local NVMe write queues, causing disk write latencies to spike from $1\text{ms}$ to $1800\text{ms}$.
4. The leader was blocked from writing heartbeat records to its WAL, causing followers to time out and trigger repeated leader election storms.

Key Lesson:

• Snapshots must be taken asynchronously using copy-on-write (fork() or memory snapshots). Never block consensus heartbeats or WAL writes on background snapshot generation.

---

6. Chapter Summary & Consensus Production Rules