Modern GPU Architecture: Streaming Multiprocessors, Tensor Cores, and VRAM Hierarchy

To engineer high-performance Large Language Model (LLM) serving systems, infrastructure engineers must understand the physical hardware silicon executing the tensor operations. Modern generative AI is not bottlenecked by general-purpose CPU compute; it is governed by the specialized microarchitecture of massively parallel accelerators—predominantly NVIDIA Hopper (H100/H200), NVIDIA Blackwell (B200), and AMD Instinct (MI300X) GPUs.

Every architectural decision in LLM serving frameworks like vLLM, TensorRT-LLM, and TGI—from PagedAttention and continuous batching to FlashAttention kernel fusion—is an explicit accommodation for the memory hierarchies, compute units, and execution models of these accelerators.

---

1. Silicon Microarchitecture: Streaming Multiprocessors (SMs)

At the silicon level, a GPU is an array of independent compute engines called Streaming Multiprocessors (SMs). The execution model is SIMT (Single Instruction, Multiple Threads).

A. The Anatomy of an SM

In the NVIDIA Hopper GH100 architecture:

• The GPU die features 132 to 144 SMs (depending on die binning and yield).
• Each SM contains:
  • 4 Warp Schedulers: Capable of dispatching instructions across warps every clock cycle.
  • 128 FP32 CUDA Cores: Standard floating-point ALU units.
  • 4 Fourth-Generation Tensor Cores: Matrix multiply-accumulate engines.
  • 256 KB Register File: Ultra-fast storage holding variable states for active threads.
  • 228 KB Unified L1 Data Cache / Shared Memory (SRAM): Software-managed on-chip scratchpad.
  • Asynchronous Data Transfer Engine (TMA - Tensor Memory Accelerator): Hardware block that transfers multidimensional tensors directly from global HBM memory into shared memory without consuming SM register or ALU cycles.

B. Warps and Warp Scheduling

Threads in CUDA are executed in locked lockstep groups of 32 threads known as a Warp.

1. Warp Divergence: If threads within the same warp take different branches of a conditional (if/else), the SM must serialize both paths, disabling inactive threads with an execution mask. This cuts execution throughput by 50% or more. High-performance LLM kernels avoid conditional branch divergence within warps.
2. Latency Hiding: GPUs lack the massive out-of-order execution logic and branch predictors found in CPUs. Instead, GPUs hide high-latency operations (like global memory loads taking 200–400 cycles) by context-switching between warps instantaneously (in 0 cycles) as long as other warps have ready instructions.

---

2. Tensor Cores: The Engine of Matrix Multiplication

Standard CUDA cores perform one floating-point scalar multiply-add operation per clock cycle ($a\times b+c$). In contrast, Tensor Cores execute matrix multiply-accumulate (MMA) operations directly at the hardware gate level in a single cycle:

$D=A\times B+C$

Where $A$ and $B$ are small sub-matrices (e.g., $16\times 16$ or $8\times 8$), and $C$ and $D$ are accumulator matrices.

Generational Evolution of Tensor Cores

---

3. The GPU Memory Hierarchy

Understanding memory latency and bandwidth across the storage tiers on the GPU die is critical for diagnosing performance bottlenecks.

Why SRAM (Shared Memory) Matters

When a CUDA kernel executes on an SM:

• Fetching data from HBM3 (Global VRAM) incurs a penalty of over 300 clock cycles.
• Fetching data from Shared Memory (SRAM) takes fewer than 30 cycles.
• The fundamental speedup of FlashAttention is moving sub-matrices of Query, Key, and Value tensors into on-chip SRAM once, computing Softmax and output reductions entirely within SRAM, and writing back to HBM only once.

---

4. Arithmetic Intensity and the Roofline Model

To predict whether an LLM workload is bottlenecked by compute units (Tensor Cores) or memory transfer speed (HBM bandwidth), engineers use Arithmetic Intensity ($I$):

$I=\frac{\text{Floating Point Operations (FLOPs)}}{\text{Memory Access (Bytes Read + Written)}}$

The Roofline Model defines the maximum attainable performance ($P$) of a GPU kernel:

$P=\min \left(P_{\text{peak}},I\times {\text{BW}}_{\text{mem}}\right)$

Where:

• $P_{\text{peak}}$ is the theoretical maximum compute throughput (e.g., $989\times 10^{12}$ FLOPs/sec for H100 FP16).
• ${\text{BW}}_{\text{mem}}$ is the memory bandwidth ($3.35\times 10^{12}$ Bytes/sec for H100 HBM3).

The Ridge Point

The Ridge Point ($I_{\text{ridge}}$) is the critical arithmetic intensity where a kernel transitions from being memory-bandwidth bound to compute bound:

$I_{\text{ridge}}=\frac{P_{\text{peak}}}{{\text{BW}}_{\text{mem}}}=\frac{989\times 10^{12}\text{ FLOP/s}}{3.35\times 10^{12}\text{ Bytes/s}}\approx 295.2\text{ FLOPs/Byte}$

• If $I<295.2$: The workload is Memory-Bound (e.g., autoregressive token generation with batch size 1 has $I\approx 1-2$). The Tensor Cores sit largely idle waiting for weights to stream from HBM.
• If $I\ge 295.2$: The workload is Compute-Bound (e.g., prompt prefill phase with batch size $\ge 32$ and context length $\ge 2048$ has $I\ge 400$). The GPU achieves maximum TFLOPS.

---

5. Hardware Interconnects: NVLink vs PCIe

When LLM weights exceed the memory capacity of a single GPU, the model must be partitioned across multiple GPUs using Tensor Parallelism (TP) or Pipeline Parallelism (PP). Inter-GPU communication bandwidth dictates the scalability of these techniques.

Bandwidth Comparison

---

6. Real-World Architecture Benchmark Script

The following Python script interrogates the local CUDA device to inspect SM topology, compute capability, memory clock rates, and theoretical peak memory bandwidth.

---

7. Production Failure Modes and Troubleshooting Runbook

Failure Mode 1: Tensor Core Inefficiency Due to Matrix Dimension Misalignment

• Symptom: Custom CUDA kernels or unpadded linear projections exhibit a 60–75% reduction in TFLOPS compared to cuBLAS benchmarks.
• Root Cause: Tensor Cores require matrix dimensions ($M,N,K$) to be exact multiples of warp tile dimensions (e.g., multiples of 8 or 16 in FP16, and multiples of 16 or 32 in FP8). If a vocabulary size or hidden dimension is not divisible by 16 (e.g., $|\mathcal{V}|=32001$), the compiler falls back to slow scalar CUDA core instructions or injects expensive boundary padding.
• Resolution: Always pad vocabulary dimensions and linear layers to the nearest multiple of 64 or 128 (e.g., expand $|\mathcal{V}|$ from $32,001$ to $32,064$).

Failure Mode 2: Multi-GPU TP Bottleneck via PCIe Fallback

• Symptom: Scaling a 70B parameter model from 1 GPU to 4 GPUs using Tensor Parallelism causes Time Per Output Token (TPOT) to increase from 30ms to 95ms instead of speeding up.
• Root Cause: The 4 GPUs are hosted on consumer or cloud PCIe slots without NVLink bridge interconnects. Tensor Parallelism executes two All-Reduce operations per transformer layer. Over PCIe Gen4 ($31.5\text{ GB/s}$), inter-GPU communication latency ($>20\mu \text{s}$) overwhelms the compute savings.
• Resolution: Restrict Tensor Parallelism ($TP>1$) exclusively to nodes with high-speed NVLink fabrics (e.g., SXM5 HGX nodes). On PCIe-only clusters, use Pipeline Parallelism or pure Data Parallel serving instead.

---

8. Summary & Architectural Key Takeaways

1. The SM is the Fundamental Compute Unit: GPU performance is a function of SM count, warp scheduler efficiency, and register occupancy.
2. Tensor Cores Multiply Matrices at the Silicon Level: FP8 and FP16 MMA operations provide 5x–10x higher compute density than traditional FP32 CUDA cores.
3. The Memory Wall Dominates Inference: HBM3 memory bandwidth ($3.35\text{ TB/s}$) is orders of magnitude slower than on-chip SRAM ($19\text{ TB/s}$). LLM generation speeds are fundamentally bounded by how fast weights and KV cache tensors can be streamed from HBM.
4. NVLink is Mandatory for Low-Latency Tensor Sharding: Distributing sub-matrices across GPUs requires hundreds of gigabytes per second of interconnect bandwidth to hide communication overhead behind computation.