The Two Phases of LLM Inference: Compute-Bound Prefill vs Memory-Bound Decode

The single most important concept in Large Language Model systems engineering is the fundamental physical asymmetry between the Prefill Phase and the Decode Phase.

Unlike traditional computer vision or embedding models that execute a single static forward pass over an input tensor, an LLM execution lifecycle is divided into two radically distinct operating regimes with opposite hardware bottlenecks:

1. The Prefill Phase (Prompt Processing): Highly parallel, compute-intensive, and Compute-Bound (Matrix-Matrix operations / GEMM).
2. The Decode Phase (Token Generation): Sequential, bandwidth-starved, and Memory-Bound (Matrix-Vector operations / GEMV).

Understanding this division—and applying the Roofline Model to analyze operational intensity—is essential for optimizing GPU utilization, batching strategies, and serving engine latency.

---

1. Phase 1: The Prefill Phase (GEMM)

When a user submits a prompt containing $N$ tokens (e.g., $N=2,048$), the LLM processes all $N$ tokens concurrently in a single forward pass.

A. Mathematical Formulation: General Matrix Multiply (GEMM)

Let $\mathbf{X}\in {\mathbb{R}}^{N\times d_{\text{in}}}$ be the input activation matrix for all prompt tokens simultaneously. The projection of these tokens against a weight matrix $\mathbf{W}\in {\mathbb{R}}^{d_{\text{in}}\times d_{\text{out}}}$ is computed as:

$\mathbf{Y}=\mathbf{XW}\text{where }\mathbf{Y}\in {\mathbb{R}}^{N\times d_{\text{out}}}$

This operation is a standard GEMM (General Matrix Multiply):

• Total Floating Point Operations (FLOPs): $2\times N\times d_{\text{in}}\times d_{\text{out}}$
• Total Weight Bytes Read from VRAM: $d_{\text{in}}\times d_{\text{out}}\times \text{bytes\_per\_param}$

B. Arithmetic Intensity of Prefill

Arithmetic Intensity ($I$) is defined as the ratio of compute operations executed per byte of data transferred from memory:

$I_{\text{prefill}}=\frac{\text{Total FLOPs}}{\text{Total Bytes Transferred}}=\frac{2\times N\times d_{\text{in}}\times d_{\text{out}}}{2\times d_{\text{in}}\times d_{\text{out}}}=N[\text{FLOPs / Byte}]$

(assuming FP16 where each weight is 2 bytes and activations are negligible compared to weights).

If the prompt length $N=2,048$, the arithmetic intensity is $\sim 2,048\text{ FLOPs/Byte}$.

Because modern GPUs (like the Nvidia H100) require an arithmetic intensity of $\ge 295\text{ FLOPs/Byte}$ to fully saturate their Tensor Cores, prompt prefill easily saturates the compute units, keeping the GPU operating at peak FLOP capacity.

---

2. Phase 2: The Decode Phase (GEMV)

Once prefill completes and the first token is generated, the model enters the sequential decode loop. To generate token $t+1$, the model passes only the single newest token ($N=1$) into the network:

${\mathbf{x}}_{\text{new}}\in {\mathbb{R}}^{1\times d_{\text{in}}}$

A. Mathematical Formulation: General Matrix-Vector (GEMV)

The forward pass is now a Matrix-Vector Multiplication (GEMV):

${\mathbf{y}}_{\text{new}}={\mathbf{x}}_{\text{new}}\mathbf{W}\text{where }{\mathbf{y}}_{\text{new}}\in {\mathbb{R}}^{1\times d_{\text{out}}}$

• Total FLOPs: $2\times 1\times d_{\text{in}}\times d_{\text{out}}$
• Total Weight Bytes Read from VRAM: $d_{\text{in}}\times d_{\text{out}}\times 2\text{ bytes}$

B. Arithmetic Intensity of Decode

$I_{\text{decode}}=\frac{2\times 1\times d_{\text{in}}\times d_{\text{out}}}{2\times d_{\text{in}}\times d_{\text{out}}}=1.0[\text{FLOP / Byte}]$

At an arithmetic intensity of $1.0\text{ FLOP/Byte}$, the GPU must read every single byte of the multi-billion parameter model from High Bandwidth Memory (HBM) into on-chip SRAM just to perform $2$ FLOPs per parameter!

---

3. The Roofline Model Analysis

The Roofline Model graphically illustrates the hardware boundaries governing these two phases:

$\text{Attainable Performance}=\min \left(\text{Peak GPU FLOP/s},\text{Memory Bandwidth (Bytes/s)}\times \text{Arithmetic Intensity}\right)$

Hardware Comparison: Nvidia H100 vs A100

During single-user decoding on an H100, over $99.6\%$ of the GPU's raw compute power sits completely idle, throttled entirely by the time required to pull weights across the VRAM memory bus.

---

4. The Maximum Token Speed Formula

We can mathematically calculate the theoretical physical speed limit for single-stream generation:

$\text{Max Generation Speed (Tokens/sec)}=\frac{\text{GPU Memory Bandwidth (Bytes/sec)}}{\text{Model Size in Bytes}}$

Case Study: Llama 3 70B in FP16 on a Single 8-GPU H100 Node

• Model Size: $70\times 10^9\text{ parameters}\times 2\text{ bytes}=140\text{ GB}$
• Total Memory Bandwidth (8 GPUs with Tensor Parallelism): $8\times 3.35\text{ TB/s}=26.8\text{ TB/s}$

$\text{Max Speed}=\frac{26,800\text{ GB/s}}{140\text{ GB}}\approx 191.4\text{ Tokens / Second}$

No algorithmic optimization, PyTorch compiler, or CUDA kernel can ever exceed this number for FP16 weights without:

1. Quantization: Compressing weights to 4-bit (AWQ/GPTQ) cuts model size to $35\text{GB}$, boosting theoretical speed to $765\text{ tok/s}$.
2. Batching: Grouping $B$ concurrent users so the $140\text{GB}$ memory read is shared across $B$ tokens simultaneously.
3. Speculative Decoding: Using a 1B draft model to guess multiple tokens per memory read.

---

5. How Continuous Batching Bridges GEMV to GEMM

When batch size increases from $B=1$ to $B=64$, the decode phase changes from Matrix-Vector multiplication back into Matrix-Matrix multiplication:

${\mathbf{X}}_{\text{batch}}\in {\mathbb{R}}^{B\times d_{\text{in}}}$

$I_{\text{decode(batch)}}=\frac{2\times B\times d_{\text{in}}\times d_{\text{out}}}{2\times d_{\text{in}}\times d_{\text{out}}}=B[\text{FLOPs / Byte}]$

At $B=64$, arithmetic intensity jumps to $64\text{ FLOPs/Byte}$, moving the decode phase up the roofline slope towards compute saturation and dramatically increasing aggregate system throughput.

---

6. Failure Modes & Production Debugging Walkthrough

Failure Mode: False "Under-Utilization" Alarms in Monitoring Dashboards

Symptom

A DevOps team inspects nvidia-smi or Prometheus GPU metrics for an inference cluster under live traffic. GPU Compute Utilization (GPU-Util) reads $18\%$, prompting the infrastructure team to downscale the instance count to save costs. Immediately upon downscaling, user latency spikes catastrophically and requests begin timing out.

Root Cause: GPU-Util Measures SM Time, Not Memory Bandwidth

nvidia-smi's GPU-Util metric reports the percentage of time that at least one CUDA kernel was active on a Streaming Multiprocessor. It does not measure Memory Bandwidth utilization. During the decode phase, the memory controller and HBM buses were $100\%$ saturated streaming weights, even though the compute SMs finished their math in $2\mu \text{s}$ and sat stalled for the remaining $10\mu \text{s}$ waiting for memory.

Production Solution

1. Monitor Memory Bandwidth Utilization via NVIDIA DCGM (Data Center GPU Manager) metrics (DCGM_FI_DEV_FB_USED and DCGM_FI_PROF_DRAM_ACTIVE).
2. Treat DRAM_ACTIVE > 85% as cluster saturation, regardless of whether compute utilization reads low.

---

7. Summary: Prefill vs Decode Comparison Matrix