Calculating KV Cache Memory Growth: Capacity Planning for Concurrency

While model weights occupy a fixed amount of VRAM regardless of traffic, the Key-Value (KV) Cache is dynamic: it expands with every newly generated token and scales linearly with the number of concurrent user sessions.

For high-throughput inference clusters serving long-context workloads (such as 32k or 128k token document summarization, code generation, and agentic workflows), KV Cache memory rapidly becomes the limiting factor that caps total system concurrency.

---

1. The Fundamental KV Cache Formula

To compute the exact bytes consumed by the KV cache for a given batch of requests:

${\text{Memory}}_{\text{KV}}=2\times L\times n_{\text{kv\_heads}}\times d_{\text{head}}\times S\times B\times b_{\text{kv}}$

Where:

• $2$: Multiplier accounting for two distinct tensors—one Key tensor ($\mathbf{K}$) and one Value tensor ($\mathbf{V}$).
• $L$: The number of Transformer layers in the model (e.g. $32$ in Llama 3 8B, $80$ in Llama 3 70B).
• $n_{\text{kv\_heads}}$: The number of Key-Value attention heads per layer.
• $d_{\text{head}}$: The hidden dimension per attention head, defined as $\frac{d_{\text{model}}}{n_{\text{query\_heads}}}$ (typically $128$).
• $S$: The total token sequence length (Prompt Tokens + Generated Tokens).
• $B$: The number of active concurrent request streams (Batch Size).
• $b_{\text{kv}}$: The storage bytes per element:
  • FP16 / BF16: $2.0\text{ bytes}$
  • FP8 (vLLM FP8 KV Cache): $1.0\text{ byte}$
  • INT4 (Experimental KV Cache): $0.5\text{ bytes}$

---

2. Attention Head Architectures: MHA vs MQA vs GQA

The evolution of attention architectures over recent years was driven primarily by the urgent need to compress the KV cache footprint.

Quantitative Comparison: KV Cache Footprint per 1,000 Tokens

Let us calculate the KV cache memory required to store 1,000 tokens for 1 user ($S=1000,B=1,b_{\text{kv}}=2$) on an 80-layer, 8192-hidden-dimension model ($d_{\text{head}}=128$):

GQA provides an $8\times$ concurrency increase over MHA without sacrificing model reasoning capabilities.

---

3. The Long Context Trap: How One Request Can Evict Fifty Users

In production serving systems with continuous batching (e.g. vLLM), all active requests share a common pre-allocated physical VRAM block pool.

Consider a cluster with an available KV Cache pool of 32 GB:

• Scenario A (Standard Chat Queries): 50 concurrent users each maintaining a 2,000-token context window ($S=2000$). $\text{Total Memory}=50\times (2\times 0.328\text{ GB})=32.8\text{ GB}(\text{System is 100\% utilized and healthy})$
• Scenario B (Long Context Ingestion): A single enterprise user submits a 64,000-token document for analysis ($S=64000$). $\text{Memory for Single User}=64\times 0.328\text{ GB}=20.99\text{ GB}$
• Because one request consumes 65% of the entire GPU's KV cache pool, the serving engine is forced to preempt and evict 32 active chat sessions, pausing their token streaming to disk or host RAM.

---

4. FP8 KV Cache Compression: Doubling Concurrency for Free

Modern serving engines (such as vLLM and TensorRT-LLM on NVIDIA Ada Lovelace, Hopper, and Blackwell) support FP8 KV Caching.

Instead of storing past Key and Value tensors in 16-bit BF16/FP16, each activation is dynamically quantized to an 8-bit floating-point representation (E4M3 or E5M2) as it is written to the cache:

$b_{\text{kv}}=1.0\text{ byte}$

${\text{Memory}}_{\text{KV (FP8)}}=\frac{1}{2}{\text{Memory}}_{\text{KV (FP16)}}$

• Memory reduction: Exactly 50%.
• Concurrency capacity: Exactly $2\times$.
• Accuracy impact: Extensive empirical benchmarks show $<0.1\%$ perplexity degradation across reasoning, coding, and math benchmarks.

---

5. Capacity Planning Script (TypeScript)

The following TypeScript utility calculates KV cache consumption, simulates memory pool exhaustion, and verifies safe concurrency limits for production deployments.

---

6. Production Failure Modes and Engineering Runbook

Failure Mode: Sudden TTFT Latency Spikes Due to KV Cache Thrashing

• Symptom: Under steady request volume, average Time to First Token (TTFT) suddenly increases from 120ms to 4,800ms, and GPU memory utilization oscillates wildly.
• Root Cause: The serving engine reached 100% KV cache allocation. When new requests arrived, the engine had to preempt active decoding requests by swapping their KV blocks from GPU HBM to CPU Host RAM (or recomputing their tokens from scratch).
• Resolution: Set an aggressive request queue limit (max_num_seqs) in the inference server and enforce a strict KV cache watermark (gpu_memory_utilization = 0.90). When the queue fills, reject incoming requests at the API Gateway with HTTP 429 rather than thrashing active VRAM.

---

7. Summary & Key Takeaways

1. KV Cache Scales Linearly with Sequence Length and Concurrency: $\text{KV Cache}=2\times L\times n_{\text{kv}}\times d_{\text{head}}\times S\times B\times b_{\text{kv}}$.
2. GQA is Essential for Modern LLMs: Grouped-Query Attention slashes KV cache memory by $8\times$ compared to legacy Multi-Head Attention.
3. Long Context Monopolizes VRAM: A single 64k or 128k prompt can consume as much KV cache as 50 standard chat requests combined.
4. FP8 KV Caching Doubles Serving Capacity: Switching from FP16 to FP8 halves KV cache footprint with negligible degradation in benchmark accuracy.