Calculating GPU Memory Footprints: Formulas for Weights, KV Cache, and Overhead

One of the most critical responsibilities of an AI Infrastructure Engineer is GPU Capacity Planning. Deploying a model without precise memory forecasting leads to catastrophic failure modes: either immediate CUDA Out of Memory (OOM) server crashes under production load, or massive over-provisioning that wastes tens of thousands of dollars per month in idle GPU cloud instances.

VRAM allocation in modern LLM serving is not static; it is composed of four distinct layers:

1. Model Parameter Weights (Static)
2. Key-Value (KV) Cache (Dynamic, scaling with concurrency and sequence length)
3. Activation Memory (Dynamic, scaling with prompt prefill batch size)
4. CUDA Runtime & Framework Overhead (Static driver buffers and memory pools)

---

1. Calculating Model Parameter Weight Memory

The static weight memory represents the baseline VRAM consumed when the model weights are loaded from disk into GPU memory before processing any user requests.

The General Formula

${\text{Memory}}_{\text{weights}}=P\times b_{\text{param}}\times (1+ϵ)$

Where:

• $P$ is the total parameter count of the model (e.g., $8\times 10^9$ for an 8B model).
• $b_{\text{param}}$ is the storage bytes per parameter, dictated by the precision:
  • FP32 (Single Precision): $4.0\text{ bytes/param}$
  • FP16 / BF16 (Half Precision): $2.0\text{ bytes/param}$
  • FP8 / INT8 (8-bit Quantized): $1.0\text{ byte/param}$
  • INT4 / AWQ / GPTQ (4-bit Quantized): $0.5\text{ bytes/param}$
• $ϵ$ is the quantization metadata overhead factor ($\approx 0.01-0.03$ for scaling factors and zero points).

Reference Weight Footprints Across Top Architectures

---

2. Calculating KV Cache Memory Growth

The Key-Value (KV) Cache stores the past attention Key and Value tensor projections for all previous tokens across all layers. As context lengths expand to 32k, 64k, and 128k tokens, KV Cache memory rapidly surpasses the model weight memory itself.

The Mathematical Formula

For a standard Multi-Head Attention (MHA) or Grouped-Query Attention (GQA) model:

${\text{Memory}}_{\text{KV}}=2\times L\times n_{\text{kv\_heads}}\times d_{\text{head}}\times S\times B\times b_{\text{kv}}$

Where:

• $2$: Multiplier accounting for two tensors (one Key tensor and one Value tensor).
• $L$: Number of transformer layers in the model.
• $n_{\text{kv\_heads}}$: Number of Key-Value attention heads.
  • In Multi-Head Attention (MHA): $n_{\text{kv\_heads}}=n_{\text{heads}}$.
  • In Grouped-Query Attention (GQA): $n_{\text{kv\_heads}}=\frac{n_{\text{heads}}}{\text{group\_size}}$ (typically 8 heads).
  • In Multi-Query Attention (MQA): $n_{\text{kv\_heads}}=1$.
• $d_{\text{head}}$: Dimension per attention head ($d_{\text{head}}=\frac{d_{\text{model}}}{n_{\text{heads}}}$).
• $S$: Total sequence length (Prompt Tokens + Generated Tokens).
• $B$: Number of concurrent active requests (Batch Size).
• $b_{\text{kv}}$: Bytes per element in the KV cache ($2\text{ bytes}$ for FP16/BF16, $1\text{ byte}$ for FP8 KV cache).

Impact of Attention Mechanisms: MHA vs GQA

Let us compare the KV cache footprint for generating a 4,096-token context across 16 concurrent users ($B=16,S=4096$) on a 70B model ($L=80,d_{\text{head}}=128$):

Standard Multi-Head Attention (MHA, $n_{\text{kv\_heads}}=64$):

${\text{Memory}}_{\text{MHA}}=2\times 80\times 64\times 128\times 4096\times 16\times 2\text{ bytes}=171.79\text{ GB}$ (Exceeds two entire 80GB GPUs just for the KV cache!)

Grouped-Query Attention (GQA, $n_{\text{kv\_heads}}=8$, as used in Llama 3):

${\text{Memory}}_{\text{GQA}}=2\times 80\times 8\times 128\times 4096\times 16\times 2\text{ bytes}=21.47\text{ GB}$ (An $8\times$ reduction in memory, fitting comfortably inside a single GPU's free cache pool!)

---

3. Activation Memory and System Overhead

A. Activation Memory

Activation memory is the temporary scratchpad VRAM required to store intermediate tensors produced during the forward pass (e.g., query-key product matrices, layer norm outputs, and feed-forward intermediate projections).

• During Token Decoding ($S=1$), activation memory is negligible ($<100\text{ MB}$).
• During Prompt Prefill ($S=8,192$), standard attention without FlashAttention consumes quadratic memory:

${\text{Memory}}_{\text{Activations (Standard)}}\approx \mathcal{O}(B\times L\times n_{\text{heads}}\times S^2)$

With FlashAttention-2 / FlashAttention-3, intermediate attention score matrices are tiled into SRAM, reducing peak activation memory to linear:

${\text{Memory}}_{\text{Activations (FlashAttention)}}\approx \mathcal{O}(B\times S\times d_{\text{model}})$

For a prompt length of $S=8,192$ with $B=1$ on Llama 3 70B, peak activation memory requires $\approx 1.5-2.5\text{ GB}$.

B. CUDA Context and Driver Overhead

When the PyTorch or CUDA runtime initializes, it allocates:

• CUDA Runtime Context: ~600 MB to 1.2 GB per GPU.
• NCCL Inter-GPU Communication Buffers: ~500 MB to 1.0 GB per GPU when using Tensor Parallelism.
• PyTorch Caching Allocator Reserved Memory: ~1.0 GB baseline.

Total static runtime overhead: $\approx 2.0-3.0\text{ GB}$ per GPU.

---

4. Production VRAM Calculation Script

Here is an enterprise-grade calculation engine written in Python that takes any Hugging Face model architecture config and computes exact capacity limits and maximum concurrent request concurrency.

---

5. Production Failure Modes and Troubleshooting Runbook

Failure Mode: CUDA OOM on Initial Batch Burst Despite Passing Weight Budget

• Symptom: The model loads into VRAM without errors, but the very first time a user submits a 16k context prompt, the server process crashes with torch.cuda.OutOfMemoryError: CUDA out of memory.
• Root Cause: The infrastructure engineer allocated 76 GB of an 80 GB GPU to model weights and KV cache pool, leaving only 4 GB for runtime operations. When the 16k prompt prefill arrived, the activation tensors during forward pass required 5.5 GB of intermediate VRAM, causing an unrecoverable out-of-memory exception.
• Resolution: In serving engines (like vLLM), set gpu_memory_utilization = 0.85 to 0.90 (never 1.0). This guarantees a 10% to 15% dedicated headroom cushion for peak prefill activations and PyTorch allocator fragmentation.

---

6. Summary & Key Takeaways

1. Total VRAM = Weights + KV Cache + Activations + Driver Overhead: Capacity planning must account for all four components simultaneously.
2. Precision Directly Scales Weight Size: FP16 uses 2 bytes/param ($140\text{ GB}$ for 70B), FP8 uses 1 byte ($70\text{ GB}$), and INT4 uses 0.5 bytes ($35\text{ GB}$).
3. Grouped-Query Attention (GQA) Saves the KV Cache: GQA reduces KV cache memory consumption by $8\times$ compared to standard Multi-Head Attention.
4. Never Allocate 100% of VRAM to Weights and Caches: Always maintain a 10–15% buffer ($8-12\text{ GB}$) on an 80GB GPU to absorb peak prompt prefill activations and CUDA context buffers.