Memory Bandwidth as the Fundamental LLM Bottleneck: HBM3e vs SRAM

A common misconception among software engineers entering the AI infrastructure space is that Large Language Model text generation is bottlenecked by the computational power (FLOPs) of the GPU. In reality, during the autoregressive token generation phase (decoding), the vast majority of the GPU's Tensor Cores sit over 80% to 90% idle.

The true, inescapable physical bottleneck of single-user LLM inference is Memory Bandwidth—the rate at which parameters and Key-Value cache tensors can be transferred from High Bandwidth Memory (HBM) into on-chip Static RAM (SRAM) and registers.

---

1. The Physics of Autoregressive Decoding: The Weight Streaming Problem

To understand why memory bandwidth is the bottleneck, examine what happens when an LLM generates a single token.

In an autoregressive transformer:

1. The user provides a prompt, which generates a hidden state vector $\mathbf{h}\in {\mathbb{R}}^{1\times d_{\text{model}}}$ representing the current token (with batch size $B=1$).
2. To compute the next token, that single vector must be multiplied by every weight matrix in the model:
   • Attention projections: ${\mathbf{W}}_q,{\mathbf{W}}_k,{\mathbf{W}}_v,{\mathbf{W}}_o$
   • Feed-forward projections: ${\mathbf{W}}_{\text{gate}},{\mathbf{W}}_{\text{up}},{\mathbf{W}}_{\text{down}}$
   • Layer normalization and LM head projection: ${\mathbf{W}}_{\text{vocab}}$
3. Every single parameter of the neural network must be loaded from HBM into the SM registers exactly once per generated token.

GEMM vs GEMV: The Matrix-Vector Arithmetic Trap

• In Matrix-Matrix Multiplication (GEMM) (used during training and prompt prefill), a matrix of shape $[B\times S,D]$ multiplies a weight matrix $[D,D^{\prime }]$. Each weight loaded from memory is reused across $B\times S$ input tokens. The arithmetic intensity is high:

$I_{\text{GEMM}}=\frac{2\times (B\times S)\times D\times D^{\prime }}{\text{Weights Loaded}+\text{Inputs}+\text{Outputs}}\approx \mathcal{O}(B\times S)$

• In Matrix-Vector Multiplication (GEMV) (used during single-token decoding with $B=1$), the input is a single row vector $[1,D]$ multiplying weight matrix $[D,D^{\prime }]$. Each weight loaded from VRAM is used exactly once for a single multiply-accumulate operation!
• The arithmetic intensity collapses:

$I_{\text{GEMV}}=\frac{2\times 1\times D\times D^{\prime }}{2\times D\times D^{\prime }\text{ (bytes)}}=1\text{ FLOP / Byte (in FP16)}$

Since an NVIDIA H100 requires an arithmetic intensity of $\approx 295\text{ FLOPs/Byte}$ to saturate its Tensor Cores, running at $1\text{ FLOP/Byte}$ means the GPU is running at less than 0.35% of its peak computational capability.

---

2. Deriving the Theoretical Maximum Generation Speed

Because every parameter must be read from VRAM for each token generated, we can derive the hard physical upper bound for token generation speed.

$\text{Time per Token (seconds)}\ge \frac{\text{Total Model Size (Bytes)}}{\text{Effective Memory Bandwidth (Bytes/sec)}}$

$\text{Tokens per Second (TPS)}\le \frac{\text{Effective Memory Bandwidth (Bytes/sec)}}{\text{Total Model Size (Bytes)}}$

Concrete Calculations Across Hardware and Models

Let us calculate the absolute theoretical limits across industry-standard GPUs and model configurations:

Case 1: Llama 3 70B (FP16: 140 GB) on 2x NVIDIA H100 SXM5 (Tensor Parallelism = 2)

• Total Weights = $70\times 10^9\times 2\text{ bytes}=140\text{ GB}$
• Combined Memory Bandwidth of 2x H100 = $2\times 3,350\text{ GB/s}=6,700\text{ GB/s}$
• Assuming 80% memory bus efficiency ($\text{Eff}=0.80$):

$\text{Max TPS}=\frac{6,700\times 0.80\text{ GB/s}}{140\text{ GB}}=\frac{5,360}{140}\approx 38.28\text{ tokens/sec}$

It is physically impossible for a single user to receive more than ~38 tokens/sec for unquantized Llama 3 70B on 2x H100s without batching or speculative decoding!

Case 2: Llama 3 70B (4-bit AWQ: 35 GB) on 1x NVIDIA H100 SXM5

• Total Weights = $70\times 10^9\times 0.5\text{ bytes}=35\text{ GB}$
• Memory Bandwidth of 1x H100 = $3,350\text{ GB/s}$
• Assuming 80% efficiency:

$\text{Max TPS}=\frac{3,350\times 0.80\text{ GB/s}}{35\text{ GB}}=\frac{2,680}{35}\approx 76.57\text{ tokens/sec}$

By quantizing weights from 16-bit to 4-bit, we reduced the bytes read per token by $4\times$, which directly translates to a $2\times -4\times$ boost in token generation speed.

---

3. Theoretical Peak Throughput Comparison Matrix

---

4. Overcoming the Memory Wall: Three Architectural Solutions

Since physical memory bandwidth is bounded by silicon manufacturing and thermal limits, how do modern LLM serving engines achieve high throughput?

A. Amortization via Batching

When batch size $B>1$, the GPU reads the weight matrix $\mathbf{W}$ from HBM once, but multiplies it against $B$ independent token vectors simultaneously.

• Memory bandwidth spent loading weights: $140\text{ GB}$ (constant regardless of batch size).
• Number of tokens generated: $B$.
• Effective memory cost per token: $\frac{140\text{ GB}}{B}$.
• At $B=64$, system throughput increases by nearly $50\times$, moving the workload into the compute-bound regime.

---

5. Memory Bandwidth Benchmarking Tool

Below is an automated diagnostic script in Python that measures actual sustainable PyTorch GPU memory bandwidth and compares empirical performance against the theoretical hardware ceiling.

---

6. Production Failure Modes and Engineering Runbook

Failure Mode: SLA Violations Due to Misconfigured Batching Timeouts

• Symptom: An inference cluster serving Llama 3 70B experiences Time to First Token (TTFT) spikes exceeding 4,000ms while GPU utilization sits at only 15%.
• Root Cause: The dynamic batching scheduler is configured with a high max_batch_delay_ms = 500ms attempting to aggregate requests for compute efficiency, but traffic volume is low. Requests sit waiting in the queue to form batches that never fill, while decoding remains memory-bandwidth starved.
• Resolution: In low-concurrency environments, switch to Continuous Batching (Iteration-Level Scheduling) with zero batch timeout (max_batch_delay = 0). Immediately begin prefill on arriving requests while interleaving active decoding iterations.

---

7. Summary & Key Takeaways

1. Decode is Memory-Bound, Prefill is Compute-Bound: Single-token generation is governed by matrix-vector operations ($I\approx 1\text{ FLOP/Byte}$), meaning performance is dictated purely by HBM bandwidth.
2. Single-Stream TPS Has a Strict Physical Limit: $\text{TPS}\le \frac{\text{Memory Bandwidth}}{\text{Model Size}}$. No software optimization can exceed this limit without reducing precision (quantization) or reducing forward passes (speculative decoding).
3. Weight Quantization Unlocks Linear Speedups: Compressing 16-bit weights to 4-bit cuts memory bus transfers by $4\times$, directly boosting single-user generation speed by up to $3\times -4\times$.
4. Concurrency Amortizes the Memory Penalty: High-concurrency continuous batching shares weight memory transfers across dozens of concurrent requests, shifting the GPU from idle memory-wait states into high-efficiency Tensor Core execution.