System Design for Real-Time LLM Inference & Streaming

The system design interview has fundamentally changed. While load balancers, database sharding, and caching layers are still relevant, the rise of Large Language Models (LLMs) has introduced entirely new architectural bottlenecks.

Designing a system to serve a typical web request (fetching JSON from a database) is a solved problem. Designing a system to generate tokens autoregressively from a 70-billion parameter model while serving thousands of concurrent users in real-time is the frontier.

In this deep dive, we will explore the core concepts of modern LLM inference architectures, focusing on memory bandwidth, KV caching, continuous batching, and token streaming.

---

1. The Core Bottleneck: Memory Bandwidth

In traditional web architectures, the bottleneck is often I/O (database lookups or network latency) or CPU (processing business logic). In LLM inference, the bottleneck is almost always Memory Bandwidth, not raw compute (FLOPs).

LLM generation happens in two distinct phases:

1. Prefill Phase (Prompt Processing): The model processes the entire input prompt at once in parallel. This phase is compute-bound. The GPU matrix multiplication units (Tensor Cores) are fully utilized.
2. Decode Phase (Token Generation): The model generates one token at a time, autoregressively. To generate a single new token, the model must read all of its weights from High Bandwidth Memory (HBM) into the GPU's compute registers. This phase is memory bandwidth-bound.

Because weights must be repeatedly loaded for every single token generated, your inference speed (tokens per second) is strictly limited by how fast your GPU can move data from its memory to its compute cores.

---

2. The Solution: KV Caching & PagedAttention

If we naively re-computed the attention scores for all previous tokens every time we generated a new token, the compute cost would scale quadratically ($O(N^2)$).

Instead, modern inference engines use KV Caching. They cache the Key (K) and Value (V) tensors computed for previous tokens. This reduces the compute complexity of generating the next token from $O(N^2)$ to $O(N)$.

The Problem with Naive KV Caching

However, managing this cache in GPU memory is incredibly difficult. Because generation lengths are unpredictable (we don't know when the model will output an <EOS> token), naive memory allocation leads to heavy fragmentation. Systems would pre-allocate maximum sequence lengths for every request, wasting up to 60-80% of GPU memory.

Enter PagedAttention (vLLM)

Inspired by virtual memory paging in operating systems, PagedAttention partitions the KV cache into fixed-size blocks (pages).

By dynamically allocating non-contiguous memory blocks, PagedAttention nearly eliminates memory fragmentation, allowing the system to fit significantly more concurrent requests into the same GPU memory, directly increasing throughput.

---

3. Continuous Batching (Iteration-Level Scheduling)

In traditional ML inference (like image classification), you batch requests together (e.g., process 16 images at once) to maximize GPU utilization.

LLM generation makes batching difficult because requests finish at different times. If you batch 4 requests, and one request finishes generating its answer in 10 tokens while the others need 500 tokens, that slot in the batch sits idle for 490 iterations.

Continuous Batching (or iteration-level scheduling) solves this. Instead of waiting for the entire batch to finish, the inference engine (like vLLM or TGI) ejects finished requests and inserts new incoming requests at the iteration level.

This ensures the GPU matrix multiplication units remain fully saturated.

---

4. The Streaming Architecture (SSE vs WebSockets)

Because token generation is slow (e.g., 20-50 tokens/second), waiting for the entire response to complete before sending it to the client results in unacceptable Time-To-First-Token (TTFT) latency.

LLM applications require streaming architectures. The two primary protocols used are Server-Sent Events (SSE) and WebSockets.

Server-Sent Events (SSE)

SSE is the industry standard for LLM streaming (used by OpenAI, Anthropic, etc.). It is a unidirectional protocol over standard HTTP/1.1 or HTTP/2.

Why SSE over WebSockets?

1. Simplicity: It operates over standard HTTP, meaning it easily passes through firewalls, proxies, and load balancers without special configuration.
2. Unidirectional: The client only needs to receive data. WebSockets provide full-duplex, bi-directional communication, which is overkill and more complex to scale for a simple request/stream-response model.

Example SSE Client Implementation:

---

5. Routing & Load Balancing (Sticky Sessions)

If you are building a ChatGPT-like application where users have long conversations, you might think you need stateful connections.

However, the standard pattern is Stateless LLM Architecture. The client (or a backend Redis cache) holds the conversation history. On every new message, the entire conversation history is sent back to the LLM.

Why? Because the LLM needs the full context to compute attention.

However, with the rise of Prompt Caching (where the KV cache of a long system prompt or document is kept alive in GPU memory across requests), Sticky Routing becomes critical.

If a user is querying a massive 100K token document, you want their subsequent requests to be routed to the exact same GPU worker that processed their first request. If they hit a different worker, the entire 100K document must be reprocessed (prefilled) from scratch, costing massive compute and TTFT latency.

Modern LLM load balancers (like Ray Serve) use session-aware routing to maximize KV cache hit rates.

---

Summary

Designing systems for the AI era requires a deep understanding of hardware bottlenecks and memory management. To build high-performance LLM infrastructure:

1. Optimize for Memory Bandwidth during decoding.
2. Use engines that support PagedAttention and Continuous Batching (vLLM, TGI, TensorRT-LLM).
3. Implement Server-Sent Events (SSE) for fast Time-To-First-Token streaming.
4. Utilize Sticky Routing to maximize KV cache reuse across conversational turns.