osFoundry / Platform
Model quantization
Quantize any model down to q4 / q3 in a few clicks. Run a 70B model on a laptop GPU.
What it is
Model quantization in osFoundry shrinks an open-weight model by lowering the numeric precision of its weights, so a model that would not fit in your GPU's memory at full precision fits after quantization. You point osFoundry at a model, pick a target level such as q4 or q3, and it produces a quantized copy you can run locally. The practical payoff is the one people search for: a 70-billion-parameter model that normally needs a multi-GPU server can be squeezed down far enough to load and run on a single laptop GPU, at the cost of some accuracy.
Because osFoundry is a local-first, bring-your-own-key desktop workspace, quantization happens on your machine and the quantized model stays on your device unless you move it. There is no cloud round-trip required and no reseller sitting between you and the model. The same workspace that runs the quantized model also runs your chat, agents, apps, code and knowledge, so a model you quantize is immediately usable as a local backend for that work rather than a file you have to wire up somewhere else.
Quantization is a trade-off, and osFoundry treats it as one you control. A q4 build keeps most of a model's quality while roughly quartering its memory footprint relative to 16-bit weights; dropping to q3 saves more memory but gives up more accuracy. You choose the level per model, keep both the original and the quantized copy, and switch between local quantized models and cloud or self-hosted models from the same interface — useful when you want a small quantized model for fast local drafts and a larger cloud model for final passes.
Key capabilities
- Quantize any open-weight model down to q4 or q3 in a few clicks, no command-line pipeline to assemble
- Run a 70B-class model on a single laptop GPU by cutting its memory footprint through quantization
- Quantization runs locally on your device and the quantized model stays local — nothing is uploaded
- Keep the original and the quantized copy side by side, and pick the precision level per model
- Run quantized local models alongside cloud and self-hosted models from one BYOK workspace, no reseller markup
Use cases
- Solo developer: quantizes a 70B model to q4 so it runs on a laptop GPU for offline coding help on a plane or in a locked-down network
- Privacy-conscious analyst: keeps a quantized model entirely on-device so sensitive documents never leave the machine during analysis
- Cost-watching builder: uses a fast local q3 model for iterative drafting and switches to a cloud model in the same workspace only for final output
Frequently asked questions
How much smaller does a model get at q4 or q3?
Quantization lowers the precision of a model's weights, so memory use drops sharply. A q4 build is roughly a quarter the size of the same model in 16-bit weights, which is why a 70B model that needs a multi-GPU server at full precision can fit on a single laptop GPU. q3 saves even more memory but gives up more accuracy, so you pick the level per model based on the trade-off you want.
Does quantizing a model reduce its quality?
Some, yes. Quantization is a trade-off: q4 keeps most of a model's quality while cutting memory substantially, while q3 saves more memory and loses more accuracy. osFoundry lets you choose the level per model and keep the original alongside the quantized copy, so you can compare and decide what is acceptable for your task rather than committing blindly.
Where does quantization run, and does my model leave my machine?
It runs locally. osFoundry is a local-first, bring-your-own-key desktop workspace, so quantization happens on your device and the quantized model stays there unless you choose to move it. There is no required cloud upload and no reseller between you and the model, which is what makes it suitable for private or offline work.