Multi-GPU LLM Scaling: Memory Limits, Interconnects, and NVLink vs PCIe

When model parameters exceed the High Bandwidth Memory (HBM) capacity of a single GPU, the model must be partitioned and distributed across a cluster of interconnected accelerators.

For example, serving the unquantized Llama 3 70B model in 16-bit precision requires $140\text{ GB}$ of static weight memory plus $30\text{ GB}$ of dynamic KV cache—far exceeding the $80\text{ GB}$ capacity of a single NVIDIA H100. Similarly, serving frontier models like Llama 3.1 405B ($810\text{ GB}$ in FP16) or DeepSeek-V3 671B requires sharding across at least 8 to 16 GPUs.

However, distributing neural network computation across multiple GPUs introduces a severe performance penalty: Inter-GPU Communication Latency. This lesson explores cluster topologies, communication bottlenecks, and the physical limits of NVLink vs PCIe.

---

1. The Scaling Threshold: When Does a Model Exceed Single-GPU Limits?

A model must be sharded across multiple GPUs whenever:

$\text{Total VRAM Required}=\text{Weight Memory}+\text{KV Cache Pool}+\text{Peak Activations}+\text{CUDA Overhead}>{\text{VRAM}}_{\text{GPU}}$

Hardware Allocation Rule of Thumb

---

2. Inter-GPU Communication: NVLink vs PCIe vs InfiniBand

When an LLM layer is sharded across GPUs, the intermediate activation tensors must be synchronized at every single transformer layer using collective communication primitives (such as All-Reduce or All-to-All).

The latency of this synchronization is governed by the physical interconnect:

Why PCIe Fails for Tensor Parallelism

In Tensor Parallelism (TP), every transformer layer executes two All-Reduce operations.

• For an 80-layer model generating 1 token: $2\times 80=160\text{ network synchronization barriers per token}$.
• Over NVLink 4 ($900\text{ GB/s},0.2\mu \text{s}$ latency): 160 synchronizations take $\approx 0.8\text{ms}$ total.
• Over PCIe Gen4 ($64\text{ GB/s},1.5\mu \text{s}$ latency): 160 synchronizations take $\approx 25\text{ms}$ total—completely destroying real-time streaming performance.

---

3. The Multi-GPU Parallelism Taxonomy

To scale across multiple GPUs and nodes, AI infrastructure relies on four distinct parallelism strategies:

---

4. Production Failure Modes and Troubleshooting Runbook

Failure Mode: NCCL Ring Deadlock During Multi-GPU Server Initialization

• Symptom: Launching a multi-GPU vLLM instance (--tensor-parallel-size 8) hangs indefinitely on startup with no error messages, and GPU utilization sits at 100% on core 0.
• Root Cause: The NVIDIA Collective Communications Library (NCCL) attempted to route inter-GPU communication over an misconfigured virtual network interface or blocked port (e.g. firewall blocking NCCL sockets).
• Resolution: Set the explicit NCCL network interface and debugging environment variables in your launch scripts:

---

5. Summary & Key Takeaways

1. Sharding is Required for Large Models: Models exceeding 80GB of combined weight and KV cache memory must be distributed across multiple accelerators.
2. Interconnect Bandwidth Dictates Performance: NVLink 4 ($900\text{ GB/s}$) provides the ultra-low latency required for intra-layer Tensor Parallelism.
3. Never Run Tensor Parallelism Over PCIe: 160 synchronization barriers per token over PCIe will degrade generation speed by up to $10\times$.
4. Combine Strategies for Scale: Frontier architectures combine Tensor Parallelism (intra-node NVLink) with Pipeline/Expert Parallelism (inter-node InfiniBand) to scale beyond 1,000 GPUs.