Understanding Core LLM Performance Metrics: TTFT, TPOT, ITL, and Throughput

Evaluating the performance of Large Language Model infrastructure requires an entirely different set of metrics than classical microservices or web applications. In traditional web services, requests are evaluated using standard End-to-End (E2E) Latency (e.g., $p99\le 100\text{ms}$).

In generative AI serving, because LLMs generate text token-by-token over prolonged streaming sessions (lasting from several hundred milliseconds to tens of seconds), single aggregate latency numbers obscure critical user-experience bottlenecks.

To design, benchmark, and scale production AI clusters, platform engineers must isolate the distinct operational dimensions: Time to First Token (TTFT), Time Per Output Token (TPOT), Inter-Token Latency (ITL), and Aggregate System Throughput.

---

1. The Core LLM Performance Metrics

A. Time to First Token (TTFT)

Time to First Token (TTFT) is the duration between the client dispatching the HTTP request and receiving the very first streaming token chunk.

$\text{TTFT}=t_{\text{network\_in}}+t_{\text{queue}}+t_{\text{prefill}}+t_{\text{first\_token\_decode}}+t_{\text{network\_out}}$

• Why it matters: TTFT dictates the perceived responsiveness of the system. In interactive user interfaces (like chatbots or code assistants), a user tolerates a slight delay before generation begins if streaming starts promptly ($<500\text{ms}$).
• Primary Bottleneck: Length of the input prompt ($N_{\text{prompt}}$) and compute availability on the GPU during the Prefill Phase.

B. Time Per Output Token (TPOT) and Inter-Token Latency (ITL)

• Inter-Token Latency (ITL): The elapsed time between the arrival of token $i$ and token $i+1$ for a specific request.
• Time Per Output Token (TPOT): The average ITL across the entire generated sequence:

$\text{TPOT}=\frac{T_{\text{E2E}}-\text{TTFT}}{N_{\text{output}}-1}$

• Why it matters: TPOT dictates the reading smoothness. Human reading speed averages $5-8$ words per second ($\sim 20-30\text{ms}$ per token). A TPOT $>80\text{ms}$ feels sluggish, while irregular ITL spikes ("token jitter") degrade interactive user experience.
• Primary Bottleneck: GPU High Bandwidth Memory (HBM) transfer speed during the Decode Phase.

C. Total End-to-End Latency ($T_{\text{E2E}}$)

The complete round-trip duration required to complete the generation request:

$T_{\text{E2E}}=\text{TTFT}+{\sum }_{i=2}^{N_{\text{output}}}{\text{ITL}}_i\approx \text{TTFT}+(N_{\text{output}}-1)\times \text{TPOT}$

D. System Throughput (Tokens per Second)

While latency measures the experience of an individual user, System Throughput measures the efficiency and capacity of the entire GPU cluster:

${\text{Throughput}}_{\text{system}}=\frac{{\sum }_{k=1}^R(N_{\text{prompt}}^{(k)}+N_{\text{output}}^{(k)})}{\Delta t}[\text{Total Tokens / Sec}]$

${\text{Throughput}}_{\text{generation}}=\frac{{\sum }_{k=1}^R{N}_{\text{output}}^{(k)}}{\Delta t}[\text{Output Tokens / Sec}]$

In production billing and capacity planning, output tokens are significantly more expensive to generate than prompt tokens due to memory-bandwidth bound single-token forward passes.

---

2. Latency vs Concurrency: The Saturation Curve

When scaling concurrent users on an inference server (such as vLLM, TensorRT-LLM, or TGI), system performance follows a characteristic saturation curve:

1. Under-Loaded Zone: A single request generates tokens at the maximum physical speed of the GPU's memory bus (e.g. $100\text{ tokens/sec}$), but GPU compute cores sit largely idle.
2. The "Knee" of the Curve: As concurrency increases, continuous batching combines multiple active requests into single tensor matrix multiplications. Output tokens per second increases by $10\times -20\times$ with negligible increase in per-user TPOT.
3. Queue Saturation Zone: When the total KV Cache requirements exceed available GPU VRAM or prefill compute is starved, requests accumulate in the server queue, causing TTFT to spike exponentially.

---

3. SLA and SLO Framework for Production AI Applications

Designing LLM architectures requires aligning infrastructure tuning with specific workload SLOs (Service Level Objectives):

---

4. Benchmark Harness Implementation in Python

Below is an automated asynchronous benchmarking script using asyncio and httpx to measure empirical $p50$, $p95$, and $p99$ for TTFT, TPOT, and Throughput against any OpenAI-compatible inference endpoint:

---

5. Failure Modes & Production Debugging Walkthrough

Failure Mode: The "Invisible Prefill Freeze" (High ITL Jitter)

Symptom

Users in an interactive chat application complain that the AI streaming text "freezes" for $2-3$ seconds in the middle of sentences, even though the average TPOT is reported as $25\text{ms}$.

Root Cause: Prefill-Decode Contention

In naive continuous batching engines, when a new user arrives with a large $16\text{k}$-token context prompt, the engine schedules the entire $16\text{k}$ prefill computation in a single massive GPU kernel execution. During those $1,700\text{ms}$, all Tensor Cores are $100\%$ occupied, completely blocking the lightweight decoding steps of all existing streaming users.

Production Solution: Chunked Prefill (Sarathi Algorithm)

Enable Chunked Prefill in the serving engine (e.g., --enable-chunked-prefill in vLLM). The server splits large prompt prefills into uniform chunks (e.g. $512$ tokens), interleaving one prefill chunk with active decode iterations on every forward pass step.

---

6. Summary: Key Metric Reference Card