Chunked Prefill: Eliminating Inter-Token Latency Jitter in Mixed Workloads

In high-concurrency LLM serving environments, inference clusters rarely process homogeneous traffic. At any given moment, a production cluster is executing two radically different computational tasks:

1. Interactive Streaming Decodes: Active user chat sessions requiring smooth, jitter-free Inter-Token Latency (ITL < 25ms) (Memory-Bandwidth Bound).
2. Long Prompt Prefills: Incoming enterprise requests submitting massive 8,000 to 64,000 token documents for processing (Compute Bound).

Without architectural isolation, an incoming 32k-token prompt prefill will monopolize the GPU for 2,000ms, causing all active streaming users to experience sudden, jarring pauses in their token output.

Chunked Prefill (introduced in the Sarathi and vLLM Chunked Prefill architectures) solves this Prefill-Decode Interference by slicing long prompt prefills into fixed-size chunks and interleaving them seamlessly with active token decode steps.

---

1. The Prefill-Decode Interference Dilemma

To understand why prompt prefills disrupt streaming decodes, compare their hardware characteristics:

When an unchunked prefill forward pass executes on the GPU:

• The entire CUDA stream is occupied running massive GEMM kernels for 1,500ms+.
• The active decoding requests cannot run their next iteration until the prefill kernel completes.
• To the human user receiving a streaming response, the AI text generation suddenly freezes and halts, destroying the interactive user experience and violating enterprise SLAs.

---

2. The Sarathi Architecture: Chunked Prefill with Piggybacked Decodes

The core insight of Sarathi (Agrawal et al., 2023) is that decoding iterations have spare compute capacity, while prefill iterations have spare memory bandwidth.

Slicing Prefill into Budgeted Chunks

Instead of processing an $N=16,384$ prompt in a single monolithic pass:

1. The scheduler divides the prompt into $K$ chunks of fixed token size $C$ (e.g. $C=512$ tokens).
2. On each iteration, the GPU schedules:
   • $512$ tokens from the incoming prefill prompt.
   • $1$ token for every active decoding stream in the batch.
3. The arithmetic intensity of the 512 prefill tokens saturates the Tensor Cores, while the decode tokens piggyback on the same forward pass with virtually zero added latency!

$\text{Tokens per Iteration}=C+B_{\text{decode}}$

Where $C$ is chosen such that the iteration duration matches the target ITL SLA (e.g., $25\text{ms}$).

---

3. Configuring Chunked Prefill in Production (vLLM)

To enable Chunked Prefill in vLLM production deployments:

Key Parameters:

• --enable-chunked-prefill: Activates Sarathi-style prefill slicing.
• --max-num-batched-tokens 512: Sets the token budget ceiling per iteration ($C$). Lowering this value (e.g. to 256 or 512) guarantees tight, jitter-free ITL at the expense of slightly longer Time to First Token (TTFT) for massive documents.

---

4. Production Failure Modes: Tuning the Chunk Size

Failure Mode: ITL Degradation Caused by Setting max_num_batched_tokens Too High

• Symptom: Interactive voice-agent conversations stutter whenever background document summarization tasks are triggered.
• Root Cause: The infrastructure engineer configured --max-num-batched-tokens = 8192 to optimize bulk throughput. An 8k token chunk takes ~350ms to compute on 4x A100s, causing ITL to spike from 25ms to 350ms on every prefill iteration.
• Resolution: For low-latency interactive applications (voice AI, real-time code autocomplete), set --max-num-batched-tokens between 256 and 512.

---

5. Summary & Key Takeaways

1. Prefill-Decode Interference Destroys User Experience: Monolithic prompt prefills block streaming decodes for seconds, creating severe latency jitter.
2. Chunked Prefill Slices Long Prompts: Splitting prompts into 512-token chunks bounds iteration latency to strict millisecond SLA budgets.
3. Decodes Piggyback for Free: Combining compute-heavy prefill chunks with memory-heavy decode steps maximizes total hardware utilization.
4. Tune max_num_batched_tokens for Your Workload: Balance raw bulk throughput with interactive streaming smoothness.