What is LLM quantization in simple terms?

It's storing a model's weights at lower numerical precision — for example 4-bit instead of 16-bit — to shrink it. Because memory scales with bits per weight, 4-bit quantization uses about four times less VRAM than FP16 and runs faster, at a small and usually acceptable cost in quality.

What does GGUF stand for and what is it?

GGUF is the single-file model format from the llama.cpp project (the successor to GGML) that bundles quantized weights with model metadata. It's optimized for CPU, Apple Silicon, and mixed CPU/GPU inference, which is why it's the standard for local tools like llama.cpp, Ollama, and LM Studio. It's a file format, not a quantization algorithm.

What's the difference between Q4, Q5, and Q8?

The number is the average bits per weight, so Q8 is larger and higher fidelity than Q5, which is larger than Q4. On a typical 8B model, Q8 is near-lossless at about 8 GB, Q5_K_M is a small step down at about 5.3 GB, and Q4_K_M is the size/quality sweet spot at about 4.6 GB with under half a percent quality loss.

Which GGUF quantization should I use?

Q4_K_M for most cases — it's the best balance of size and quality and the standard starting point. Step up to Q5_K_M or Q8_0 if you have spare VRAM and want higher fidelity, and only go below Q4 (using importance-matrix IQ quants) when memory is genuinely tight, since quality drops sharply at 3-bit and below.

How much quality do you lose with 4-bit quantization?

Surprisingly little. On a peer-reviewed Llama-3.1-8B evaluation, Q4_K_M scored under half a percent below the full-precision baseline on average benchmarks. The noticeable degradation starts at 3-bit, especially on reasoning and math tasks. Exact numbers vary by model, but 4-to-5-bit is widely considered the safe zone.

GGUF vs GPTQ vs AWQ — which should I pick?

GGUF for local, laptop, and Apple Silicon inference via llama.cpp or Ollama. GPTQ and AWQ store a GPU-friendly layout and are the better choice for GPU-server throughput with vLLM or TGI — AWQ is faster to produce, GPTQ is often the most accurate weight-for-weight. bitsandbytes is the easiest since it quantizes at load time with no calibration.

How do I calculate the VRAM a quantized model needs?

Use bytes ≈ (parameters × bits-per-weight) ÷ 8. At 4-bit that's roughly 0.5 to 0.6 GB per billion parameters, so an 8B model is about 4.5 GB. Then add headroom for the KV cache and roughly 10 to 15% framework overhead — the running footprint is always larger than the file on disk.

← Resources

By Ren Sugaya — Performance Engineer, GPU & Inference · 2026-05-26 · INSIGHT

LLM Quantization Explained: GGUF, Q4 vs Q8, and Quality

Quantization shrinks an LLM by storing its weights at lower precision, cutting VRAM roughly fourfold at 4-bit. This guide explains the VRAM math, decodes GGUF quant names like Q4_K_M, compares GGUF to GPTQ and AWQ, and shows where quality actually breaks down.

What quantization is, and why it matters

Quantization reduces the numerical precision used to store model weights — typically from FP16 (16 bits, 2 bytes per weight) down to 8-bit or 4-bit. Because memory scales linearly with bits per weight, 4-bit uses about four times less VRAM than FP16. It also speeds up inference, since token generation is bound by memory bandwidth, not just compute.

The size math is simple: bytes ≈ (parameters × bits-per-weight) ÷ 8. An 8B model at FP16 is about 16 GB; at 4-bit it's roughly 4.5 GB. That difference is what turns a model that needs a data-center GPU into one that runs on a laptop.

GGUF is a file format, not a method

GGUF is a single-file format that stores quantized weights plus model metadata (tokenizer, architecture, hyperparameters). It's the successor to GGML from the llama.cpp project, optimized for CPU, Apple Silicon (Metal), and mixed CPU/GPU offload — which is why it dominates local and laptop inference through llama.cpp, Ollama, and LM Studio.

The quant level lives in the filename. The number is the average bits per weight; `_K` means k-quants (the modern super-block method with double-quantized scaling factors and layer-aware bit allocation); `_S`, `_M`, `_L` are small, medium, and large variants. `_M` selectively keeps sensitive layers — attention and output — at higher precision, which is why Q4_K_M beats a flat 4-bit quant.

Q4 vs Q5 vs Q8: size and quality, with numbers

A peer-reviewed evaluation on Llama-3.1-8B makes the trade-offs concrete (sizes are for that model; ratios generalize). Against an FP16 baseline perplexity of about 7.32 (lower is better):

Q8_0 — 8.5 bits/weight, ~7.95 GiB, perplexity 7.33: essentially lossless.
Q6_K — 6.56 bpw, ~6.14 GiB, perplexity 7.35.
Q5_K_M — 5.70 bpw, ~5.33 GiB, perplexity 7.40.
Q4_K_M — 4.89 bpw, ~4.58 GiB, perplexity 7.56: the practical sweet spot, under half a percent off the baseline.
Q3 and below — sharp, consistent degradation, especially on reasoning tasks.

The 4-to-5-bit band is the safe zone. Below 3-bit you need importance-matrix (IQ) quants, which use calibration data to spend precision where it matters most — usable, but a last resort for tight memory.

GGUF vs GPTQ vs AWQ

GGUF isn't the only option; the right one depends on where you run.

GGUF: built for CPU, Mac, and mixed offload. Pick it for local and desktop inference.
GPTQ: post-training, calibration-based, uses second-order information to minimize per-layer error. Most accurate weight-for-weight at a given bit width, but slow to produce and oriented toward GPU serving (vLLM, TGI, TensorRT-LLM).
AWQ: protects the roughly 1% most salient weights based on activation magnitudes; faster to produce than GPTQ and GPU-optimized at 4-bit.
bitsandbytes: quantizes on the fly at load time with no pre-quantized checkpoint or calibration — the easiest path, widely used for QLoRA fine-tuning.

The practical split: GGUF for local and laptop, AWQ or GPTQ for GPU-server throughput.

How much quality do you actually lose?

Less than most people fear at 4-bit, and more than they expect below 3-bit. On the Llama-3.1-8B evaluation, Q8 was within rounding of full precision, Q4_K_M lost under half a percent on average benchmarks, and the cliff appeared at 3-bit, with the clearest degradation on reasoning-heavy evaluations like grade-school math. Remember that absolute perplexity and size numbers are model-specific — the percentages generalize, but always check the figures for your base model. And budget VRAM beyond the file size: the KV cache and roughly 10–15% framework overhead need headroom too.

Quantization in osFoundry

Choosing a quant normally means understanding bits-per-weight, leaving room for the KV cache, and hunting through a wall of cryptic GGUF suffixes to find the one file that fits without crashing. osFoundry's local inference removes that step: it detects your available memory and auto-selects the right quantization — Q4_K_M for tight VRAM, Q5_K_M or Q8_0 when there's headroom — so the model runs at the best quality your hardware supports. The catalog surfaces only compatible builds instead of every variant, and falls back gracefully if a heavier quant won't fit. You get the size-versus-quality sweet spot of hand-picking a GGUF without needing to know what Q4_K_M means.

Frequently asked questions

What is LLM quantization in simple terms?: It's storing a model's weights at lower numerical precision — for example 4-bit instead of 16-bit — to shrink it. Because memory scales with bits per weight, 4-bit quantization uses about four times less VRAM than FP16 and runs faster, at a small and usually acceptable cost in quality.
What does GGUF stand for and what is it?: GGUF is the single-file model format from the llama.cpp project (the successor to GGML) that bundles quantized weights with model metadata. It's optimized for CPU, Apple Silicon, and mixed CPU/GPU inference, which is why it's the standard for local tools like llama.cpp, Ollama, and LM Studio. It's a file format, not a quantization algorithm.
What's the difference between Q4, Q5, and Q8?: The number is the average bits per weight, so Q8 is larger and higher fidelity than Q5, which is larger than Q4. On a typical 8B model, Q8 is near-lossless at about 8 GB, Q5_K_M is a small step down at about 5.3 GB, and Q4_K_M is the size/quality sweet spot at about 4.6 GB with under half a percent quality loss.
Which GGUF quantization should I use?: Q4_K_M for most cases — it's the best balance of size and quality and the standard starting point. Step up to Q5_K_M or Q8_0 if you have spare VRAM and want higher fidelity, and only go below Q4 (using importance-matrix IQ quants) when memory is genuinely tight, since quality drops sharply at 3-bit and below.
How much quality do you lose with 4-bit quantization?: Surprisingly little. On a peer-reviewed Llama-3.1-8B evaluation, Q4_K_M scored under half a percent below the full-precision baseline on average benchmarks. The noticeable degradation starts at 3-bit, especially on reasoning and math tasks. Exact numbers vary by model, but 4-to-5-bit is widely considered the safe zone.
GGUF vs GPTQ vs AWQ — which should I pick?: GGUF for local, laptop, and Apple Silicon inference via llama.cpp or Ollama. GPTQ and AWQ store a GPU-friendly layout and are the better choice for GPU-server throughput with vLLM or TGI — AWQ is faster to produce, GPTQ is often the most accurate weight-for-weight. bitsandbytes is the easiest since it quantizes at load time with no calibration.
How do I calculate the VRAM a quantized model needs?: Use bytes ≈ (parameters × bits-per-weight) ÷ 8. At 4-bit that's roughly 0.5 to 0.6 GB per billion parameters, so an 8B model is about 4.5 GB. Then add headroom for the KV cache and roughly 10 to 15% framework overhead — the running footprint is always larger than the file on disk.