Home / Glossary / Quantization
What is Quantization?
Quantization reduces an LLM’s memory footprint by storing weights in lower precision (4-bit / 5-bit / 6-bit) instead of full 16-bit floats. osFoundry quantises models in one click to Q4_K_M, Q5_K_M, Q6_K, or FP16.
Detail
Quantization changes the precision of each weight, not the number of weights — a 7B model stays a 7B model with coarser numbers. LLM weights carry enough redundancy that the output distribution barely moves at moderate precision loss; push too far and output visibly degrades.
In the GGUF ecosystem you will see levels like Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_S: the digit is the approximate bit width, K variants allocate extra precision to important weights, and S/M/L tune that allocation.
Memory shrinks roughly with bit width. A 7B model needs about 14 GB in FP16, ~7.5 GB at Q8_0, ~4.5 GB at Q4_K_M; a 70B model still exceeds 40 GB even at Q4. Weights aside, the KV cache needs its own memory at runtime.
The community rule of thumb: Q8_0 is indistinguishable in practice, Q4_K_M is the standard quality/memory sweet spot, Q3 and below degrade visibly. Sensitivity is task-dependent — code generation suffers first, summarization holds up longest — and smaller models degrade earlier than larger ones at the same level. Start at Q4_K_M and adjust: loosen quantization or move up a size class if quality falls short. Related techniques include KV-cache quantization and QLoRA (fine-tuning on quantized weights).
How osFoundry approaches Quantization
osFoundry’s install flow lets you pick the quantisation at install time. Swap between quantisations later without retraining — same model handle, different backing weights.
Frequently asked questions
What does Q4_K_M mean?
A ~4-bit K-type mixed quantization with a medium (M) precision allocation. It is the standard quality/memory trade-off most local-LLM users start with.
Does quantization hurt quality?
Q8 is nearly indistinguishable from FP16, Q4_K_M is the accepted sweet spot, and Q3 and below degrade visibly. Sensitivity is task-dependent — code generation suffers first, summarization holds up longest.
How does quantization affect file size?
Model files shrink roughly in proportion to bit width: Q8 is about half of FP16, Q4 about a quarter — download time and disk usage shrink the same way.
Related terms
Related features