Understanding Network Latency, Bandwidth, and Throughput

In single-machine computing, reading from L1 cache takes $\approx 1\text{ns}$ and reading from RAM takes $\approx 100\text{ns}$. In distributed systems, communication is bounded by the speed of light in fiber optics and the physical constraints of networking hardware.

To design low-latency, high-throughput distributed systems, you must understand the mathematical relationships governing Latency, Bandwidth, and Throughput.

---

1. The Three Fundamental Dimensions

[!IMPORTANT] Analogy:
If a network is a highway:

• Latency is the time it takes for a single car to drive from City A to City B (speed limit).
• Bandwidth is the number of lanes on the highway (width).
• Throughput is the actual number of cars that arrive per minute (flow rate).

---

2. Deconstructing Latency: The Four Component Delays

The end-to-end latency of a packet is the sum of four distinct physical delays:

$L_{\text{total}}=D_{\text{propagation}}+D_{\text{transmission}}+D_{\text{processing}}+D_{\text{queuing}}$

1. Propagation Delay ($D_{\text{propagation}}$)

The time taken for a bit of signal to travel along the physical medium (copper cable or silica fiber).

• Light in a vacuum travels at $300,000\text{ km/s}$.
• Light in fiber glass travels at $\approx 200,000\text{ km/s}$ ($\approx 5\mu \text{s}$ per kilometer).
• For a round-trip between New York and London ($\approx 11,000\text{ km}$ fiber path), the theoretical minimum physical RTT is: ${\text{RTT}}_{\text{min}}=\frac{11,000\text{ km}}{200,000\text{ km/s}}\approx 55\text{ms}$ No software optimization or caching algorithm can ever reduce this physical constant.

2. Transmission (Serialization) Delay ($D_{\text{transmission}}$)

The time required for the network interface card (NIC) to push all bits of a packet onto the physical wire: $D_{\text{transmission}}=\frac{\text{Packet Size (bits)}}{\text{Link Bandwidth (bps)}}$

• Pushing a $1.5\text{KB}$ MTU packet onto a $10\text{Mbps}$ link takes $1.2\text{ms}$.
• Pushing the same $1.5\text{KB}$ packet onto a $100\text{Gbps}$ datacenter link takes only $0.00012\text{ms}$ ($120\text{ns}$).

3. Processing Delay ($D_{\text{processing}}$)

The time taken by intermediate routers and destination hosts to inspect the packet header, verify the IP/TCP checksum, and determine the output routing interface (typically $<10\mu \text{s}$).

4. Queuing Delay ($D_{\text{queuing}}$)

The time a packet spends waiting in router buffer queues before being serialized onto the outgoing link. If incoming traffic exceeds the link rate, buffer queues fill up, causing queuing delay to spike exponentially from microseconds to hundreds of milliseconds.

---

3. The Bandwidth-Delay Product (BDP)

The Bandwidth-Delay Product (BDP) defines the maximum amount of data that can be in flight across the network at any given instant:

$\text{BDP}=\text{Bandwidth}\times \text{Round-Trip Time (RTT)}$

Real-World Example:

Consider a high-speed $10\text{Gbps}$ fiber link between San Francisco and Frankfurt ($\text{RTT}=150\text{ms}$):

$\text{BDP}=10\text{ Gbps}\times 0.150\text{ s}=1.5\text{ Gigabits}=187.5\text{ Megabytes}$

If your TCP socket buffer (Receive Window) is configured to the default of $64\text{KB}$, the sender can only transmit $64\text{KB}$ before pausing and waiting $150\text{ms}$ for an ACK:

$\text{Max Throughput}=\frac{64\text{ KB}}{0.150\text{ s}}\approx 426\text{ KB/s}(0.004\%\text{ of link capacity!})$

To achieve the full $10\text{Gbps}$ throughput over high-latency links, TCP Window Scaling (SO_RCVBUF and SO_SNDBUF) must be configured to match or exceed the BDP.

---

4. Little's Law in Distributed Systems

In queueing theory and distributed systems architecture, Little's Law relates concurrency, throughput, and latency:

$L=\lambda \times W$

Where:

• $L$ = Number of concurrent in-flight requests (System Concurrency / In-Flight Load)
• $\lambda$ = Arrival rate / Throughput (Requests Per Second)
• $W$ = Average response time / Latency (Seconds per Request)

Architectural Implications:

1. If latency doubles ($W:100\text{ms}\to 200\text{ms}$) while incoming traffic ($\lambda$) remains constant at $5,000\text{ RPS}$, the number of concurrent open connections ($L$) doubles from $500$ to $1,000$.
2. If connection pool limits are fixed ($L_{\text{max}}=500$), an increase in downstream service latency immediately throttles system throughput (${\lambda }_{\text{max}}=\frac{L_{\text{max}}}{W}$), dropping throughput from $5,000\text{ RPS}$ to $2,500\text{ RPS}$ and generating $503\text{ Service Unavailable}$ errors.

---

5. Code Deep-Dive: Measuring RTT vs. Server Execution Time

In distributed tracing and RPC debugging, client-measured latency includes both network transit time and server execution time. To detect network degradation, instruments must isolate transit delay:

---

6. Production Failure Postmortem: Tail Latency Amplification

When a single user request fans out to $N$ downstream services in parallel, system latency is bounded by the slowest response:

${\text{Latency}}_{\text{system}}=\max (L_1,L_2,\dots ,L_N)$

If individual service requests have a $99^{\text{th}}$ percentile ($p99$) latency of $100\text{ms}$ (a $1\%$ probability of being slow):

Mitigation Techniques:

• Hedged Requests: Send a duplicate request to a second replica if the first has not responded within the $p95$ threshold ($15\text{ms}$), accepting whichever returns first.
• Tied Requests: Enqueue requests on multiple replicas simultaneously and cancel redundant executions once one begins processing.

---

7. Chapter Summary & Formula Reference