Disk I/O Fundamentals and the Operating System Page Cache

Every database engine is ultimately bounded by the physics of physical storage media and the operating system kernel I/O subsystems.

Understanding the difference between sequential I/O and random I/O, the mechanics of the Linux Page Cache, and the cost of fsync() is essential to designing high-throughput distributed storage systems.

---

1. The Physics of Storage: Sequential vs Random I/O

Why are database storage engines specifically structured around append-only logs and sorted pages?

The SSD Block Erase Penalty:

Even on modern NVMe SSDs, flash memory cannot overwrite data in place:

1. Data is written in Pages ($4\text{KB}-16\text{KB}$).
2. Data can only be erased in large Blocks ($2\text{MB}-8\text{MB}$).
3. Random updates force the SSD controller to perform Garbage Collection and Write Amplification, degrading drive longevity and creating unpredictable p99 latency spikes.
4. Sequential writes allow the SSD controller to stream data directly into contiguous flash blocks with zero write amplification.

---

2. The Linux Page Cache and Dirty Pages

When an application calls write() in Linux, data is NOT immediately written to physical disk.

Instead, the kernel copies the byte buffer into the Linux Page Cache (unallocated DRAM used by the kernel) and marks the memory page as Dirty.

The Background pdflush / flusher Threads:

The Linux kernel periodically flushes dirty pages in the background according to sysctl kernel parameters:

• vm.dirty_background_ratio (default 10%): When dirty pages exceed 10% of total RAM, background threads begin writing to disk.
• vm.dirty_ratio (default 20%): If dirty pages reach 20%, all application write() calls block until dirty pages are flushed.

---

3. The True Cost of fsync() and Group Commit

Because the OS Page Cache is volatile, an operating system crash or server power cut will destroy all dirty pages.

To guarantee ACID Durability, databases must call fsync(fd) or fdatasync(fd).

However, fsync() blocks execution until the drive controller confirms that data is physically persistent on flash NAND cells:

• On enterprise NVMe SSDs: A single fsync() takes $0.5\text{ms}-2.0\text{ms}$.
• Calling fsync() synchronously after every single query limits database write throughput to $\approx 500-1000\text{ writes/sec}$!

The Solution: Group Commit

High-performance storage engines (PostgreSQL, MySQL InnoDB, RocksDB) batch concurrent client writes into a single shared fsync() execution:

---

4. Bypassing the Kernel: Direct I/O (O_DIRECT)

Database engines (like MySQL InnoDB and ScyllaDB) often maintain their own user-space Buffer Pools with sophisticated LRU page eviction policies.

Having both the database buffer pool and the Linux Page Cache store the same data in DRAM causes Double Buffering (wasting 50% of available server RAM).

To avoid this, enterprise databases open files with the O_DIRECT flag:

• Bypasses the Linux Page Cache entirely.
• Transfers data directly between user-space memory buffers and the NVMe drive using Direct Memory Access (DMA).
• Prevents cache pollution and reduces CPU context switching overhead.

---

5. Code Deep-Dive: Write Throughput vs fsync Microbenchmark

---

6. Production Failure Postmortem: The Synchronous Logging Bottleneck

Incident Overview:

In 2022, a high-frequency fintech payment gateway experienced a 90% throughput collapse during a sudden traffic spike, with CPU utilization sitting idle at only 8%.

What Happened:

1. An engineer enabled synchronous audit logging to satisfy a compliance requirement, invoking fsync() synchronously after every transaction record.
2. Under 5,000 requests/sec, client worker threads spent 92% of their runtime blocked in kernel fsync wait queues.
3. The NVMe SSD queue depth reached maximum saturation, causing request latencies to degrade from $3\text{ms}$ to $1400\text{ms}$.

Remediation:

• Implemented Asynchronous Group Commit with an in-memory ring buffer flushed by a dedicated writer thread every $10\text{ms}$ or every 500 records.
• Restored gateway throughput to over 25,000 transactions/sec.

---

7. Chapter Summary & Disk I/O Architecture Rules