Memory Fragmentation in Naive LLM Serving Systems

Prior to the introduction of modern memory-managed serving engines like vLLM and TensorRT-LLM, deploying Large Language Models in production was plagued by massive memory inefficiency. Early serving frameworks (such as naive Hugging Face Pipelines, FasterTransformer, and standard PyTorch serving wrappers) suffered from severe Memory Fragmentation, routinely wasting 60% to 80% of available GPU VRAM.

As a result, an 80GB GPU that theoretically possessed enough memory to serve 50 concurrent users would crash with Out-of-Memory (OOM) errors with fewer than 10 active requests.

---

1. The Twin Evils: Internal vs External Fragmentation

In standard computing systems, memory fragmentation takes two forms. In naive LLM serving systems, both forms compound to destroy GPU utilization.

A. Internal Fragmentation: The Static Over-Allocation Trap

Because the exact number of tokens a model will generate is unknown upfront (it depends dynamically on when the model emits an <|endoftext|> token or hits a stop word), naive systems pre-allocate a contiguous chunk of VRAM sized for the maximum possible sequence length (e.g., $S_{\max }=2048$ or $4096$).

• If User A asks: "What is the capital of France?"
• Prompt = 8 tokens. Generated answer = 4 tokens ("The capital of France is Paris."). Total = 12 tokens.
• Memory allocated: 4,096 tokens.
• Internal Fragmentation Waste: $\frac{4096-12}{4096}=99.7\%\text{ wasted memory}$!

Across a fleet of users, the average request length in production chat systems is only 300–600 tokens. Pre-allocating for 4k or 8k means over 85% of allocated KV cache memory sits completely empty.

B. External Fragmentation: Dynamic Allocation Pitfalls

To avoid static over-allocation, some early systems attempted to reallocate dynamic tensor buffers on the fly: starting with 64 tokens, then calling torch.cat() or re-allocating when the buffer filled up.

This created severe External Fragmentation:

1. When Request A terminates, its memory block is returned to the PyTorch caching allocator.
2. When Request B terminates, another non-adjacent memory block is freed.
3. Over time, VRAM becomes a checkerboard of small, disconnected free segments.
4. When a new request arrives requiring a contiguous 2 GB block, the allocation fails because no single contiguous block exists, even if 15 GB of total free space is scattered across hundreds of holes.

---

2. Memory Utilization in Naive Serving: The Empirical Reality

Empirical benchmarking from the UC Berkeley Sky Computing lab (the creators of vLLM) revealed the true breakdown of memory utilization in naive systems:

• Actual Useful KV Cache Data: Only 20.4% of VRAM.
• Wasted via Fragmentation: 66.5% of VRAM.
• Serving Efficiency: The system achieved only $\frac{1}{4}$ of its potential user concurrency.

---

3. The Inability to Share Memory Across Requests

Beyond fragmentation, naive contiguous memory allocation made it impossible to implement advanced inference optimizations:

A. Parallel Sampling and Beam Search

When generating multiple candidate completions for a single prompt ($N=4$ branches):

• Naive systems must duplicate the entire prompt KV cache 4 times in separate contiguous memory spaces.
• If the system prompt is 2,000 tokens, 8,000 tokens worth of identical KV data are written to VRAM.

B. Shared System Prompts

Enterprise applications frequently prepend identical 1,500-token system instructions (role definitions, guardrails, and few-shot examples) to every user prompt. In naive serving, this system prompt is independently recomputed and stored in contiguous memory for every single concurrent user.

---

4. Production Failure Modes and Engineering Runbook

Failure Mode: Premature OOM Under Moderate Concurrency

• Symptom: An internal Hugging Face Text Generation Inference (TGI) or PyTorch deployment with 8 concurrent users crashes with OutOfMemoryError, while nvidia-smi shows only 45 GB of 80 GB VRAM utilized immediately prior to the crash.
• Root Cause: The serving engine uses static contiguous array allocation for KV caches with max_sequence_length = 4096. When the 9th request arrived, the allocator could not find a contiguous block large enough to fit a full 4k-token buffer, triggering an unrecoverable OOM.
• Resolution: Migrate the serving infrastructure from naive PyTorch/HuggingFace generation to a paged virtual-memory serving engine (vLLM or TensorRT-LLM).

---

5. Summary & Key Takeaways

1. Static Allocation Wastes 80% of VRAM: Pre-allocating contiguous memory for the maximum possible sequence length creates massive internal fragmentation for short and medium-length queries.
2. Dynamic Reallocation Creates External Fragmentation: Reallocating tensors on every token step causes memory checkerboarding, where small unusable holes prevent new requests from allocating.
3. No Memory Sharing: Contiguous allocation prohibits sharing identical prompt KV states across parallel sampling branches or enterprise system prompts.
4. The Need for Virtual Memory: The fatal flaws of contiguous memory allocation led directly to the invention of PagedAttention, which applies Operating System virtual memory paging to LLM KV caches.