What is Quantization?
Quantization reduces AI model precision to make them smaller and faster. Learn how it enables running large language models on consumer hardware.
Quantization converts the high-precision numbers that represent a model's learned knowledge into lower-precision formats. A model trained with 32-bit floating point weights might be quantized to 8-bit or even 4-bit integers, reducing memory requirements by 4-8x. This tradeoff enables running 70B parameter models on a single GPU that would otherwise require server clusters.
Why it matters
Quantization is what makes the current explosion of local AI deployment possible. Without it, running sophisticated language models would remain the exclusive domain of companies with six-figure infrastructure budgets. For marketers and business teams, this creates new options: run AI locally for data privacy, deploy in environments without reliable internet, or simply reduce ongoing API costs. A company using quantized local models for internal content analysis might save $10,000+ monthly compared to cloud APIs while keeping sensitive data on-premise. As models grow larger and more capable, quantization techniques will remain essential for making cutting-edge AI accessible beyond the biggest tech companies.
What people get wrong
- Quantization always significantly degrades model quality. 8-bit quantization typically shows less than 3% performance drop on standard benchmarks. For many business applications, this is imperceptible. Heavy 2-bit quantization does hurt quality substantially, but moderate quantization is often free from practical impact.
- You need special expertise to use quantized models. Pre-quantized models are available for most popular open-source models. Loading them requires the same code as full-precision models with minor configuration changes. Libraries like llama.cpp and vLLM handle the technical details automatically.
- Quantization only matters for local deployment. Cloud providers also use quantization to reduce inference costs and latency. When you pay per token for API calls, the provider's infrastructure efficiency directly affects pricing. Quantized serving powers many commercial API offerings.
Related terms
Model Parameters: Quantization directly affects how much memory each parameter consumes, determining whether large parameter counts are practically deployable.
Inference: Quantization accelerates inference speed by reducing memory bandwidth requirements and enabling more efficient computation paths.
Open Source AI: Quantization enables running open-source models like Llama locally, which is a primary reason for choosing open weights over API-only services.
Frequently Asked Questions
What is quantization in AI?
Quantization is a compression technique that reduces the numerical precision of AI model weights. Instead of storing weights as 32-bit floating point numbers, they're converted to 8-bit, 4-bit, or even lower precision formats. This makes models 4-8x smaller and faster while maintaining most of their capabilities.
How much accuracy do you lose with quantization?
8-bit quantization typically shows less than 3% performance drop on benchmarks. 4-bit quantization may cause 5-15% degradation depending on the task. Simple tasks like classification are barely affected, while complex reasoning sees more impact. Testing on your specific use case is essential.
What's the difference between GPTQ and AWQ quantization?
GPTQ (GPT-Quantization) focuses on minimizing reconstruction error layer by layer during compression. AWQ (Activation-aware Weight Quantization) preserves weights that have high activation values more carefully. AWQ often produces slightly better results for equivalent bit widths, but GPTQ has wider tooling support.
Can I quantize any AI model?
Most transformer-based models can be quantized using standard tools. You need the original weights in a compatible format (typically PyTorch or Safetensors). Proprietary models like GPT-4 or Claude cannot be quantized since their weights aren't publicly available. Quantization tools like llama.cpp, GPTQ, and bitsandbytes handle common architectures automatically.
What hardware do I need to run quantized models?
A 4-bit quantized 7B parameter model runs on GPUs with 6GB VRAM, like an RTX 3060. A 70B model at 4-bit needs roughly 35GB, requiring an A100 40GB or multiple consumer GPUs. CPU-only inference is possible with llama.cpp but significantly slower than GPU execution.
Is quantization the same as model distillation?
No. Quantization compresses an existing model by reducing numerical precision. Distillation trains a smaller model to mimic a larger one's outputs. Distillation creates genuinely smaller architectures with fewer parameters. Quantization keeps the same architecture but stores weights more efficiently.