Post-Training Quantization (PTQ) vs Quantization-Aware Training (QAT)

Quantization is the process of mapping high-precision, continuous floating-point weights and activation values into lower-bit discrete representations (such as 8-bit or 4-bit integers).

By quantizing an LLM, infrastructure teams can compress a 70B parameter model from 140 GB down to 35 GB, enabling it to run on a single 80GB GPU while accelerating memory bandwidth streaming by up to $3\times -4\times$.

However, mapping millions of continuous floating-point numbers into a tiny set of 16 discrete integer levels (in 4-bit quantization) inevitably introduces Quantization Error. This lesson explores the two primary methodologies for quantizing deep neural networks: Post-Training Quantization (PTQ) and Quantization-Aware Training (QAT).

---

1. The Mathematical Foundation: Uniform Linear Quantization

Quantization maps a continuous real-valued tensor $\mathbf{X}\in \mathbb{R}$ into an integer tensor $\mathbf{Q}\in \mathbb{Z}$ through a Scale Factor ($S$) and an optional Zero-Point ($Z$).

A. Uniform Asymmetric Quantization

In asymmetric quantization, the real value range $[\alpha ,\beta ]$ is mapped to the integer range $[q_{\min },q_{\max }]$:

$S=\frac{\beta -\alpha }{q_{\max }-q_{\min }}$

$Z=\text{round}\left(-\frac{\alpha }{S}\right)+q_{\min }$

$\mathbf{Q}=\text{clamp}\left(\text{round}\left(\frac{\mathbf{X}}{S}\right)+Z,q_{\min },q_{\max }\right)$

B. Dequantization (Reconstruction)

During the forward pass, the integer values are reconstructed back to floating-point approximations:

$\hat{\mathbf{X}}=S\times (\mathbf{Q}-Z)\approx \mathbf{X}$

C. Uniform Symmetric Quantization

In symmetric quantization, the zero-point is constrained to zero ($Z=0$), mapping $[-\alpha ,\alpha ]$ symmetrically around zero:

$S=\frac{\max (|\mathbf{X}|)}{q_{\max }}$

$\mathbf{Q}=\text{clamp}\left(\text{round}\left(\frac{\mathbf{X}}{S}\right),-q_{\max },q_{\max }\right)$

$\hat{\mathbf{X}}=S\times \mathbf{Q}$

Symmetric quantization simplifies GPU matrix multiply-accumulate hardware logic because zero-point subtraction is eliminated.

---

2. Weight-Only vs Weight-Activation Quantization

Depending on whether activations are also quantized, quantization schemes fall into two architectural categories:

Key Differences:

1. Weight-Only (W4A16 / W8A16):
   • Targets the memory bandwidth bottleneck of the decoding phase.
   • Weights are stored in 4-bit on HBM ($4\times$ memory compression), but dequantized into FP16 inside on-chip SRAM registers before computation.
   • Extremely easy to calibrate; virtually zero accuracy degradation on reasoning tasks.
2. Weight-Activation (W8A8 / FP8):
   • Quantizes both weights and incoming token activations to 8-bit.
   • Enables the GPU to use native INT8 / FP8 Tensor Cores, doubling theoretical computational throughput (TFLOPS) for prompt prefill.
   • More vulnerable to activation outlier spikes.

---

3. The Role of Calibration Datasets in PTQ

Unlike simple scalar rounding, Post-Training Quantization uses a Calibration Dataset (typically 128 to 512 representative text sequences from Wikipedia, RedPajama, or C4) to determine the optimal clipping thresholds:

1. MinMax Calibration: Sets clipping boundaries to the absolute minimum and maximum observed values. Vulnerable to outlier distortion.
2. Mean Squared Error (MSE) Calibration: Iterates through candidate clipping thresholds $\alpha$ to minimize the reconstruction error: ${\min }_{\alpha }\Vert \mathbf{X}-\hat{\mathbf{X}}(\alpha ){\Vert }_2^2$
3. KL Divergence (KLD) Calibration: Uses Kullback-Leibler divergence to match the probability distribution of quantized activations with the original FP16 distribution.

---

4. Production Failure Modes: Calibration Data Bias

Failure Mode: Severe Reasoning Degradation Caused by Domain Mismatch in Calibration

• Symptom: A 70B coding model quantized with INT4 PTQ shows normal perplexity on Wikipedia benchmarks, but fails completely on generating valid Python and SQL syntax.
• Root Cause: The calibration dataset used during the PTQ pipeline consisted entirely of conversational general-knowledge English text without any source code samples. The quantization scaler clipped high-frequency programming punctuation tokens (such as {, }, _, ->), distorting code embedding projections.
• Resolution: Always ensure calibration datasets represent a balanced distribution of target production domains (including code, math formulas, markdown tables, and multi-turn conversations).

---

5. Summary & Key Takeaways

1. Quantization Compresses Weights and Memory Bandwidth: Mapping FP16 to 4-bit integers reduces model size by $4\times$, unlocking massive throughput boosts.
2. PTQ Requires Zero Retraining: Post-Training Quantization calibrates scaling factors using 128–512 sample prompts in minutes on a single GPU.
3. Weight-Only vs Weight-Activation: W4A16 accelerates memory-bound decoding; W8A8 / FP8 accelerates compute-bound prompt prefilling.
4. Calibration Data Matters: Use representative domain data during PTQ calibration to prevent clipping critical token activations.