Fundamentals of Numerical Precision: FP32, FP16, BF16, and FP8 Explained

In deep learning and Large Language Model infrastructure, numerical precision dictates three critical dimensions of performance: VRAM memory consumption, memory bandwidth transfer time, and Tensor Core computational throughput (TFLOPS).

Transitioning a 70B parameter model from traditional 32-bit floating-point (FP32) to modern 8-bit floating-point (FP8) reduces its memory footprint from 280 GB to 70 GB—allowing the model to fit on a single NVIDIA H100 GPU while quadrupling inference throughput.

However, reducing bit precision introduces severe numerical instability challenges: underflow, overflow, and quantization noise. This lesson explores the IEEE 754 binary anatomy of floating-point representations and explains why BF16 and FP8 dominate modern AI silicon.

---

1. IEEE 754 Binary Anatomy: Exponents vs Mantissas

Every floating-point binary representation encodes a real number $x$ using three discrete bit fields:

$x=(-1{)}^{\text{sign}}\times 2^{\text{exponent}-\text{bias}}\times \left(1+\frac{\text{mantissa}}{2^{\text{num\_mantissa\_bits}}}\right)$

1. Sign Bit ($S$): 1 bit determining whether the number is positive ($0$) or negative ($1$).
2. Exponent Bits ($E$): Dictates the Dynamic Range (the scale between the smallest non-zero number and the largest representable number).
3. Mantissa / Fraction Bits ($M$): Dictates the Precision / Resolution (the number of significant decimal digits of accuracy).

---

2. FP16 vs BF16: Why BF16 Replaced FP16 in Modern LLMs

When half-precision (16-bit) training and inference was first introduced, models relied on standard FP16 (IEEE 754).

The Exponent Mismatch in FP16

In deep transformer networks with 80+ layers, attention logits ($\mathbf{Q}{\mathbf{K}}^T$) and SwiGLU activations frequently exceed $65,504$. In FP16, any number $>65,504$ immediately overflows to +inf, causing the entire forward pass to produce NaN (Not a Number) tokens.

Google's Bfloat16 (BF16) Solution

Developed by Google Brain for TPUs and adopted by NVIDIA Ampere/Hopper GPUs:

• BF16 preserves the full 8-bit exponent of FP32, matching its exact dynamic range ($\approx 10^{-38}$ to $10^{38}$).
• BF16 trades off mantissa bits ($7$ bits vs $10$ bits in FP16), sacrificing fine-grained fractional precision in exchange for bulletproof numerical stability.

---

3. The FP8 Era: E4M3 vs E5M2 on Hopper and Blackwell

NVIDIA Hopper (H100) and Blackwell (B200) architectures introduced native hardware FP8 Tensor Cores, enabling 8-bit floating-point matrix multiplications at up to 2x the throughput of BF16.

The FP8 specification (standardized by the Open Compute Project by NVIDIA, ARM, and Intel) defines two distinct formats:

Numerical Comparison Across All Standard AI Data Types

---

4. Production Failure Modes: Diagnosing Precision Collapse

Failure Mode: NaN Tokens Caused by FP16 Dynamic Range Overflow

• Symptom: During generation, the model suddenly outputs repeated ! ! ! ! or unprintable Unicode characters, and logs show loss = NaN.
• Root Cause: The model was loaded with torch_dtype=torch.float16 on an unscaled RoPE embedding model. High-frequency positional rotations caused query-key dot products to exceed $65,504$, overflowing the 5-bit FP16 exponent.
• Resolution: Always load modern LLMs (Llama 3, Mistral, Qwen) using torch_dtype=torch.bfloat16.

---

5. Summary & Key Takeaways

1. Exponent Determines Range, Mantissa Determines Precision: Exponent bits prevent underflow/overflow; mantissa bits preserve fractional accuracy.
2. BF16 is the Industry Standard for 16-Bit: By retaining the 8-bit exponent of FP32, BF16 eliminates the catastrophic overflow errors inherent to FP16.
3. FP8 Halves Memory and Doubles TFLOPS: Hopper and Blackwell Tensor Cores execute FP8 E4M3 and E5M2 operations at up to $2,000$ TFLOPS per GPU.
4. E4M3 for Weights, E5M2 for Gradients: Use higher mantissa precision (E4M3) for inference forward passes, and higher dynamic range (E5M2) for gradients and KV caches.