How Large Language Models Process Text: The Transformer Architecture and Autoregressive Decoding

At the core of modern Generative Artificial Intelligence sits the Autoregressive Decoder-Only Transformer. While deep learning historically relied on Recurrent Neural Networks (RNNs) and Long Short-Term Memory (LSTM) networks that processed tokens sequentially, the breakthrough of the Transformer architecture eliminated sequential recurrence during training by introducing Multi-Head Self-Attention.

However, during inference (text generation), the model must still generate output sequentially—one token at a time. Each generated token is appended to the context window and fed back into the model to predict the next token.

Understanding the exact physical mechanics of this forward pass, the internal tensor transformations, and the mathematical constraints of autoregressive generation is the foundational prerequisite for designing high-throughput LLM serving infrastructure.

---

1. The Anatomy of the Transformer Decoder

Modern production LLMs (such as Llama 3, Mistral, DeepSeek, and GPT-4) are standardly built as Decoder-Only Transformers. Unlike the original 2017 "Attention Is All You Need" encoder-decoder architecture designed for machine translation, decoder-only models use a unified causal stack that handles both prompt comprehension and text generation.

A. Token Embedding and Coordinate Space

When raw text is passed to an LLM, it is first mapped into discrete integer IDs by a subword tokenizer (such as Byte-Pair Encoding):

$\mathbf{x}=[t_1,t_2,\dots ,t_N],t_i\in \{0,1,\dots ,|\mathcal{V}|-1\}$

where $|\mathcal{V}|$ is the vocabulary size (e.g., $128,256$ in Llama 3).

The embedding lookup table is a learnable matrix:

${\mathbf{W}}_e\in {\mathbb{R}}^{|\mathcal{V}|\times d_{\text{model}}}$

Each token integer $t_i$ indexes a continuous vector ${\mathbf{e}}_i\in {\mathbb{R}}^{d_{\text{model}}}$ (where $d_{\text{model}}$ typically ranges from $4,096$ in 8B models to $8,192$ in 70B models).

B. Positional Encoding: Rotary Position Embeddings (RoPE)

Because self-attention operations are inherently permutation-invariant (order-agnostic), the model requires explicit positional information.

Early transformers used absolute sinusoidal embeddings or learned absolute positional tables. Modern production models utilize Rotary Position Embeddings (RoPE) (Su et al., 2021). RoPE encodes relative position by rotating query and key vectors in the 2D complex plane:

For a 2D component of query vector ${\mathbf{q}}_m$ at sequence index $m$ and key vector ${\mathbf{k}}_n$ at sequence index $n$:

${\mathbf{R}}_{\Theta ,m}^d{\mathbf{q}}_m=\left(\begin{array}{cc}\cos m{\theta }_i& -\sin m{\theta }_i\\ \sin m{\theta }_i& \cos m{\theta }_i\end{array}\right)\left(\begin{array}{c}{q}_m^{(2i)}\\ q_m^{(2i+1)}\end{array}\right)$

The inner product between a rotated query and a rotated key depends strictly on their relative distance $(m-n)$:

$⟨{\mathbf{R}}_m\mathbf{q},{\mathbf{R}}_n\mathbf{k}⟩={\mathbf{q}}^T{\mathbf{R}}_{m-n}\mathbf{k}$

This property enables models to generalize to sequence lengths far beyond their initial training context through techniques like RoPE frequency scaling (YaRN).

---

2. Multi-Head and Grouped-Query Self-Attention

The core computational engine of the transformer layer is the Self-Attention Mechanism.

A. Standard Scaled Dot-Product Attention

Given an input activation matrix $\mathbf{X}\in {\mathbb{R}}^{S\times d_{\text{model}}}$ (where $S$ is sequence length), we compute Query ($\mathbf{Q}$), Key ($\mathbf{K}$), and Value ($\mathbf{V}$) projections:

$\mathbf{Q}=\mathbf{X}{\mathbf{W}}_q,\mathbf{K}=\mathbf{X}{\mathbf{W}}_k,\mathbf{V}=\mathbf{X}{\mathbf{W}}_v$

where ${\mathbf{W}}_q,{\mathbf{W}}_k,{\mathbf{W}}_v\in {\mathbb{R}}^{d_{\text{model}}\times d_{\text{model}}}$.

The causal attention score matrix $\mathbf{A}$ is computed as:

$\mathbf{A}=\text{Softmax}\left(\frac{\mathbf{Q}{\mathbf{K}}^T}{\sqrt{d_k}}+\mathbf{M}\right)$

$\text{Output}=\mathbf{AV}{\mathbf{W}}_o$

where $\mathbf{M}$ is the Causal Attention Mask:

${\mathbf{M}}_{i,j}=\{\begin{array}{ll}0& \text{if }i\ge j\\ -\infty & \text{if }i<j\end{array}$

The causal mask ensures that token $i$ can only attend to previous tokens $j\le i$, preventing "future information leakage" during autoregressive generation.

B. Attention Variants: MHA vs MQA vs GQA

Serving multi-billion parameter models in production requires managing the GPU memory overhead of the Key-Value (KV) Cache. The architecture of the attention heads dictates this memory consumption:

1. Multi-Head Attention (MHA): Each Query head has an independent Key and Value head ($H_Q=H_{KV}$). Found in GPT-3 and original Llama 1.
2. Multi-Query Attention (MQA): All Query heads share a single Key and single Value head ($H_{KV}=1$). Reduces KV Cache size by a factor of $H_Q$ (e.g. $32\times$), but can cause slight accuracy degradation.
3. Grouped-Query Attention (GQA): Query heads are partitioned into $G$ groups, with each group sharing one Key and one Value head (e.g., $H_Q=32,H_{KV}=8$). Standard in Llama 2/3, Mistral, and Gemma.

---

3. Feed-Forward Networks and Activation Functions (SwiGLU)

Following the self-attention block and a residual connection, activations pass through a Feed-Forward Network (FFN). The FFN acts as the model's primary associative memory store, where factual knowledge and logical relationships are encoded into matrix weights.

While classical transformers used a two-layer MLP with ReLU or GELU activations, modern LLMs use SwiGLU (Swish Gated Linear Unit):

$\text{SwiGLU}(\mathbf{x})=\left(\text{Swish}(\mathbf{x}{\mathbf{W}}_{\text{gate}})\odot (\mathbf{x}{\mathbf{W}}_{\text{up}})\right){\mathbf{W}}_{\text{down}}$

where:

• $\text{Swish}(z)=z\cdot \sigma (\beta z)=\frac{z}{1+e^{-\beta z}}$
• $\odot$ represents element-wise Hadamard multiplication.
• ${\mathbf{W}}_{\text{gate}},{\mathbf{W}}_{\text{up}}\in {\mathbb{R}}^{d_{\text{model}}\times d_{\text{ffn}}}$
• ${\mathbf{W}}_{\text{down}}\in {\mathbb{R}}^{d_{\text{ffn}}\times d_{\text{model}}}$
• In Llama 3, $d_{\text{ffn}}\approx \frac{8}{3}{d}_{\text{model}}$ (e.g. $14,336$ for an $8\text{B}$ model).

---

4. The Autoregressive Generation Loop in Code

Below is a complete, minimal, and fully runnable Python implementation of an autoregressive forward pass and generation loop using pure NumPy:

---

5. Logit Post-Processing and Sampling Strategies

The raw output of the final linear projection layer (the LM Head) is a vector of unnormalized real numbers called logits $\mathbf{z}\in {\mathbb{R}}^{|\mathcal{V}|}$. Converting these logits into diverse, coherent text requires specific sampling parameters:

A. Temperature ($T$)

Temperature scales the logit vector before the Softmax function is applied:

$P(w_i)=\frac{\exp (z_i/T)}{{\sum }_{j=1}^{|\mathcal{V}|}\exp (z_j/T)}$

• $T\to 0$ (Greedy Decoding): The highest logit collapses to probability $1.0$, while all other probabilities become $0.0$. Deterministic, factual, but susceptible to repetitive loops.
• $T=1.0$: Standard model probability distribution.
• $T>1.0$: Flattens the distribution, increasing entropy and creative randomness, but risking nonsensical hallucinations.

B. Nucleus Sampling (Top-$P$)

Instead of restricting selection to a fixed number of tokens, Top-$P$ (Nucleus Sampling) samples from the smallest set of tokens whose cumulative probability exceeds threshold $P$:

${\sum }_{i\in {\mathcal{V}}^{(P)}}P(w_i)\ge P$

• In high-confidence contexts (e.g., following "The capital of France is"), the top-1 token " Paris" may have a probability of $0.98$, so the candidate pool dynamically shrinks to $1$ token.
• In open-ended contexts, the candidate pool dynamically expands to dozens of plausible tokens.

C. Top-$K$ and Min-$P$ Sampling

• Top-$K$: Truncates the sampling pool strictly to the top $K$ tokens with highest logits, setting all other logits to $-\infty$.
• Min-$P$: Sets a dynamic threshold relative to the top token's probability: $\text{threshold}=p_{\text{max}}\times \text{min\_p}$. If $p_{\text{max}}=0.8$ and $\text{min\_p}=0.05$, any token with probability $<0.04$ is pruned. Min-$P$ frequently outperforms Top-$P$ on modern models.

---

6. Failure Modes & Production Debugging Walkthrough

Failure Mode 1: Degenerate Looping Under Greedy Sampling ($T=0$)

Symptom

A customer support chatbot gets stuck in an infinite repetition loop: "Please provide your account number. Please provide your account number. Please provide your account number..."

Root Cause

Under greedy sampling ($T=0$), if the model assigns a slightly higher probability to repeating a phrase than concluding the thought, appending that phrase back into the context increases the attention weight on that exact phrase. On the next step, the probability of repeating the phrase increases further, creating a self-reinforcing deterministic loop.

Production Solution

1. Introduce a Repetition Penalty ($\alpha \in [1.1,1.2]$) or Frequency / Presence Penalty in the inference engine.
2. Set a low non-zero temperature ($T=0.2-0.4$) combined with $Top\text{-}P=0.9$ to break deterministic feedback cycles.
3. Configure stop_sequences or repetition detection hooks at the API gateway layer to terminate generation when an $N$-gram loop threshold is breached.

---

Failure Mode 2: Premature Termination via Special Token Misinterpretation

Symptom

An inference server abruptly terminates output after 3 words on valid user prompts, returning an empty or half-sentence response with finish_reason: "stop".

Root Cause

The tokenizer mapped user input or internal prompt templates directly to an unescaped EOS (End-Of-Sequence) token ID (e.g., <|im_end|>, <|endoftext|>, or ID 128001). When the model encounters this token in its output distribution, the serving runtime immediately stops generation.

Production Solution

1. Explicitly configure the tokenizer to treat special tokens as text literals unless inserted by the trusted system prompt wrapper:
   python

   2 linesCopy

   Loading code editor...
2. Validate that the serving engine's stop_token_ids configuration matches the exact base model tokenizer metadata.

---

7. Summary & Architectural Takeaways