Home / Features / Train and fine-tune / Quantisation

Quantise models for cheap inference on osFoundry

osFoundry quantises any open-weight model down to Q4_K_M, Q5_K_M, Q6_K, or FP16 in one click. A 70B model quantised to Q4 fits on a single A100 80 GB or two consumer 24 GB cards — running at 5–10× lower cost than FP16 with minimal quality loss for chat workloads.

Quick answer

One-click quantisation: Q4_K_M, Q5_K_M, Q6_K, FP16.
Quantised models keep the same routing handle — drop-in replacement.
5–10× cost reduction at Q4 for typical chat tasks.
Run quality benchmarks against your own eval set before swapping.

Frequently asked questions

How much quality is lost at Q4?

For chat models on chat tasks, ~1–3 percentage points on standard benchmarks. Code-generation models lose more — try Q5_K_M for code.

Can I revert if quality drops?

Yes. Quantised models are added alongside the original — never replace it.

Related features