LLM Serving Engine Shootout: vLLM vs TensorRT-LLM vs SGLang vs llama.cpp

LLM Serving Engine Shootout: vLLM vs TensorRT-LLM vs SGLang vs llama.cpp

The economics and performance of generative AI at scale are governed by the efficiency of the LLM Serving Engine. Deploying modern Large Language Models (such as Llama 3 70B, DeepSeek-V3, Mixtral 8x22B, and Qwen 2.5) in production environments is not merely a matter of executing matrix multiplications on a GPU. It is an intricate operating-systems challenge involving dynamic GPU memory management, continuous request scheduling, KV cache prefix reuse, and hardware-specific kernel fusion.

Inference operates in two fundamentally distinct computational regimes:

1. Prefill Phase (Prompt Processing): Compute-bound ($O(N^2)$ attention compute over $N$ prompt tokens, saturating GPU Tensor Cores).
2. Decode Phase (Token Generation): Memory bandwidth-bound ($O(1)$ autoregressive generation per step, constrained by GPU High Bandwidth Memory / HBM throughput).

Over the past two years, four dominant engines have emerged to solve these hardware bottlenecks:

• vLLM: The pioneer of PagedAttention, eliminating memory fragmentation and standardizing continuous batching.
• TensorRT-LLM: NVIDIA’s compiler-driven enterprise engine with Fused Multi-Head Attention (FMHA), Deep Learning accelerator optimization, and maximum raw throughput on NVIDIA Hopper/Blackwell GPUs.
• SGLang: The specialized runtime featuring RadixAttention (compressed prefix-tree caching) for complex multi-turn conversations, agent workflows, and structured JSON output.
• llama.cpp: The minimalist, dependency-free C/C++ engine dominating CPU, Apple Silicon Metal, and edge hardware via GGUF k-quants.

This architectural deep dive analyzes the inner mechanics, memory layouts, cache eviction graphs, and kernel optimizations of each engine, concluding with mathematical memory bounds, latency benchmarks, and production deployment blueprints.

---

---

1. The Physics of LLM Inference: Memory Bandwidth vs Compute

To understand why traditional deep learning frameworks (like raw PyTorch eager execution) fail at production serving, we must examine the Roofline Model of GPU execution.

1.1 The Arithmetic Intensity Gap

Arithmetic intensity ($I$) is the ratio of computational operations (FLOPs) to memory traffic (Bytes transferred):

$I=\frac{\text{Total Operations (FLOPs)}}{\text{Total Memory Access (Bytes)}}$

• Prefill Phase: When processing an initial prompt of $N=2048$ tokens, the model computes all activations simultaneously. The compute scales quadratically with sequence length while weights are loaded only once. Arithmetic intensity is high ($I\gg 100\text{ FLOPs/Byte}$), fully saturating NVIDIA H100 Tensor Cores ($989\text{ TFLOPS}$ FP16).
• Decode Phase: To generate a single token ($N=1$), the GPU must read the entire model weight tensor ($70\text{ billion parameters}\times 2\text{ bytes}\approx 140\text{ GB}$) and the accumulated KV cache from HBM into SRAM registers just to compute a single forward pass. Arithmetic intensity collapses ($I\approx 1\text{ FLOP/Byte}$).

On an NVIDIA H100 GPU with $3.35\text{ TB/s}$ HBM3 bandwidth, the maximum theoretical decode speed for an unquantized 70B model with a batch size of 1 is physically bounded by:

$\text{Max Generation Rate}=\frac{3,350\text{ GB/s}}{140\text{ GB/token}}\approx 24\text{ tokens/second}$

To achieve high throughput, the engine must batch multiple concurrent requests together ($B>1$), reusing the loaded model weights across $B$ tokens simultaneously.

---

2. The KV Cache Bottleneck: Mathematics & Memory Fragmentation

The Key-Value (KV) Cache avoids recomputing keys and values for historical tokens at every autoregressive step. However, the KV cache grows dynamically with sequence length and rapidly consumes the majority of GPU VRAM.

2.1 KV Cache Memory Equation

For a Transformer model using Multi-Head Attention (MHA), the KV cache memory required per token per request is:

$\text{BytesPerToken}=2\times 2\times n_{\text{layers}}\times n_{\text{heads}}\times d_{\text{head}}\times \text{BytesPerElement}$

Where:

• $2$: Represents the separate Key and Value tensors.
• $2$: Represents standard FP16 / BF16 floating-point precision ($2\text{ bytes}$).
• $n_{\text{layers}}$: Number of transformer decoder layers.
• $n_{\text{heads}}$: Number of key-value attention heads (reduced in Grouped-Query Attention / GQA).
• $d_{\text{head}}$: Dimension per head ($d_{\text{model}}/n_{\text{heads}}$).

For Llama 3 70B ($n_{\text{layers}}=80$, $n_{\text{kv\_heads}}=8$, $d_{\text{head}}=128$, using BF16):

$\text{BytesPerToken}=4\times 80\times 8\times 128\times 2=655,360\text{ bytes}\approx 640\text{ KB per token}$

For a batch of $B=64$ concurrent requests with a context length of $L=8192$ tokens:

$\text{Total KV Cache}=64\times 8192\times 640\text{ KB}\approx 335.5\text{ GB of VRAM}$

This surpasses the total memory of four $80\text{ GB}$ NVIDIA H100 GPUs solely for intermediate attention state.

2.2 The Memory Fragmentation Crisis

In naïve serving architectures (such as standard HuggingFace Accelerate or basic PyTorch pipelines), memory for the maximum possible sequence length ($L_{\text{max}}$) must be allocated contiguously upfront:

Contiguous pre-allocation results in three catastrophic forms of waste:

1. Reserved Over-allocation: Memory reserved for future tokens that the user may never generate.
2. Internal Fragmentation: Unused slots within pre-allocated static buffers.
3. External Fragmentation: Free memory scattered in non-contiguous fragments too small to satisfy new incoming requests.

---

3. Deep Dive: vLLM & PagedAttention

Created by Woosuk Kwon et al. at UC Berkeley (2023), vLLM revolutionized LLM infrastructure by introducing PagedAttention, an algorithm directly inspired by the virtual memory and paging subsystems of classic operating systems.

3.1 PagedAttention Mechanics

Instead of requiring contiguous physical memory, PagedAttention partitions the KV cache of each sequence into fixed-size Physical Blocks (typically 16 or 32 tokens per block).

• Logical Blocks: The sequence view seen by the attention computation.
• Physical Blocks: Allocated non-contiguously on-demand from a global GPU memory pool.
• Block Table: Maps logical block indices to physical GPU memory addresses.

During attention computation, the custom CUDA PagedAttention kernel dynamically gathers non-contiguous key-value blocks using the block table:

$A_{i,j}=\frac{{\mathbf{q}}_i\cdot {\mathbf{k}}_j^T}{\sqrt{d_{\text{head}}}},\text{where }{\mathbf{k}}_j\text{ is fetched via }\text{BlockTable}[\lfloor j/B_{\text{size}}\rfloor ]$

3.2 Copy-on-Write (CoW) for Parallel Sampling & Beam Search

When an application requests multiple completions from a single prompt (e.g., temperature sampling with $n=4$ or tree-of-thought search), multiple output sequences share the identical prompt prefix.

Through Copy-on-Write, shared prefix memory overhead drops to zero, enabling massive batch scaling for synthetic data generation and reasoning exploration.

---

4. Deep Dive: SGLang & RadixAttention

While vLLM solved intra-request fragmentation, modern AI architectures (e.g., autonomous coding agents, multi-turn chat, few-shot prompt chaining, and tree search) exhibit massive inter-request prefix sharing.

Created by Lianmin Zheng et al. (2024), SGLang introduces RadixAttention, a runtime that treats the entire GPU KV cache as a dynamic Radix Tree (Compressed Prefix Trie).

4.1 Radix Tree Prefix Cache Mechanics

In traditional engines, once a request finishes, its KV cache is freed. If the user replies in a multi-turn conversation, the engine must recompute the prefill for all historical tokens.

In SGLang:

1. When a sequence completes, its physical KV pages are retained in GPU VRAM and indexed in a radix tree where tree edges represent token sequences.
2. When a new request arrives, SGLang performs a longest-prefix match on the radix tree.
3. If a match of length $K$ is found, the engine completely skips prefill for the first $K$ tokens, reducing Time to First Token (TTFT) from hundreds of milliseconds to under $5\text{ ms}$.

4.2 LRU Eviction with Reference Counting

When GPU memory reaches capacity, SGLang evicts leaf nodes using an eviction policy based on Least Recently Used (LRU) tracking and reference counting:

4.3 Fast Constrained Decoding with Finite State Machines

For structured generation (JSON schemas, regex validation), SGLang compiles the regex into an optimized Finite State Machine (FSM) via Outlines. At each decode step, the FSM masks invalid token logits directly in GPU memory before softmax, ensuring $100\%$ valid JSON generation without CPU sampling roundtrips.

---

5. Deep Dive: TensorRT-LLM (NVIDIA Enterprise Engine)

Developed directly by NVIDIA engineers, TensorRT-LLM is an enterprise-grade inference compiler and execution runtime tailored specifically for NVIDIA GPU architectures (Ampere, Ada Lovelace, Hopper H100/H200, and Blackwell B200).

5.1 Fused Multi-Head Attention (FMHA) & Kernel Fusion

Standard PyTorch executes attention via separate kernel launches:

1. Matrix multiplication ($Q\times K^T$).
2. Softmax scaling & normalization.
3. Dropout/masking.
4. Matrix multiplication ($A\times V$).

Between each step, intermediate tensors are written back to slow global HBM memory. TensorRT-LLM compiles these operations into a single, fused FMHA CUDA kernel that executes entirely within on-chip SRAM (Shared Memory / L1 Cache), completely bypassing HBM for intermediate attention scores (delivering $2.5\times \text{–}4\times$ speedups over unfused kernels).

5.2 Native In-Flight Batching (IFB)

TensorRT-LLM implements iteration-level scheduling natively in C++:

• Rather than waiting for an entire batch to finish generating, newly arrived requests are injected into the active batch at the very next decode iteration.
• Completed requests are evacuated immediately, eliminating idle GPU bubbles.

---

6. Deep Dive: llama.cpp & GGUF Quantization

Developed by Georgi Gerganov, llama.cpp takes the diametrically opposite architectural approach: maximum portability, zero external dependencies, and extreme optimization for commodity hardware (Apple Silicon Metal, x86 AVX-512, ARM NEON, and consumer GPUs).

6.1 GGUF File Format & mmap() Instant Startup

GGUF is a self-contained, binary container format that stores all model weights, tokenizer metadata, and hyperparameter tensors aligned to physical memory page boundaries (e.g., $32\text{ bytes}$).

• When llama.cpp loads a $40\text{ GB}$ GGUF model, it does not read the bytes through standard file I/O into heap memory.
• It invokes mmap(), mapping the file directly into the process virtual address space.
• Model startup takes less than 10 milliseconds, allowing models to be loaded and unloaded dynamically on resource-constrained servers.

6.2 The Anatomy of K-Quants (Q4_K_M, Q5_K_M, IQ4_XS)

Standard uniform quantization (like vanilla INT4) applies a single scaling factor across an entire matrix, resulting in catastrophic perplexity degradation on delicate attention projection layers.

llama.cpp uses k-quants (k-means quantization), breaking weight matrices into super-blocks (e.g., 256 weights) subdivided into 8 mini-blocks (32 weights each). Different layers receive tailored precision:

---

7. The Comprehensive Benchmark Shootout

7.1 Architecture & Feature Comparison Matrix

7.2 Decision Matrix: Which Engine Should You Deploy?

---

8. Production Python Benchmarking Suite

The following asynchronous Python load generator measures Time to First Token (TTFT), Inter-Token Latency (ITL), and Total Request Throughput across OpenAI-compatible serving endpoints (vLLM, SGLang, TensorRT-LLM Triton, llama.cpp server):

---

9. Production SRE Deployment Runbooks

9.1 vLLM Production Deployment (Docker with Multi-GPU Tensor Parallelism)

9.2 SGLang Production Deployment (High-Concurrency Prefix Caching)

---

10. Academic Bibliography & Further Reading

1. Kwon, W., Li, Z., Zhuang, S., Sheng, Y., Zheng, L., Yu, C. H., Gonzalez, J. E., Zhang, H., & Stoica, I. (2023). Efficient Memory Management for Large Language Model Serving with PagedAttention. In Proceedings of the 29th ACM Symposium on Operating Systems Principles (ACM SOSP '23). https://arxiv.org/abs/2309.06180
2. Zheng, L., Yin, L., Xie, Z., Sun, C., Huang, J., Yu, C. H., Cao, S., Kozyrakis, C., Stoica, I., Gonzalez, J. E., & Sheng, Y. (2024). SGLang: Efficient Execution of Structured Language Model Programs. https://arxiv.org/abs/2312.07104
3. Dao, T., Fu, D. Y., Ermon, S., Rudra, A., & Ré, C. (2022). FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness. In Advances in Neural Information Processing Systems (NeurIPS 2022). https://arxiv.org/abs/2205.14135
4. Frantar, E., Ashkboos, S., Hoefler, T., & Alistarh, D. (2022). GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
5. Gerganov, G. (2023). llama.cpp: Port of Facebook's LLaMA model in C/C++. GitHub Repository. https://github.com/ggerganov/llama.cpp
6. InitNode Signal Deep Dive: High-Throughput Linux I/O: io_uring & eBPF/XDP vs epoll. InitNode Signal.