osFoundry / Platform
Local servers
Run local inference on your machine or workstation. Routes are automatic — Maestro picks local when it can.
What it is
Local servers let you run AI inference on your own machine instead of sending every request to a hosted API. osFoundry ships a local inference server that loads GGUF model files from a folder on your Mac or workstation and serves them to the rest of the app. Because the model runs on hardware you already own, a local run costs nothing per token, needs no API key, and keeps the prompt and its data on your device — which matters when the content is private, regulated, or simply not something you want leaving the building.\n\nInside osFoundry, Maestro is the routing layer that decides where each request goes. Routes are automatic: Maestro picks the local server whenever a request fits what your machine can handle, and reaches for a cloud model (using your own key) only when a task needs capabilities the local model can't cover. You don't wire this up per message — it's a standing policy across chat, agents, and pipeline roles. You keep control over the local runtime from the settings panel: how many transformer layers to offload to the GPU, the context window size, the KV-cache quantization (f16, bf16, q8_0, q4_0 and others), the number of parallel request slots, and, on multi-GPU machines, how to split the model across devices.\n\nThis matters because most \"AI in your workspace\" products give you exactly one place data can go: their servers. osFoundry is bring-your-own-key and local-first by design. Local servers are the floor of that design — the option that costs nothing, exposes nothing, and works offline for the models your hardware can run — while your BYOK cloud keys sit right beside them for the heavier jobs. One workspace, both paths, no per-seat fee and no reseller markup on either.
Key capabilities
- Runs GGUF models directly on your Mac or workstation through a built-in llama.cpp inference server — no cloud round-trip, no API key needed for local runs
- Maestro routes each turn automatically and picks the local server whenever a request fits, falling back to a cloud model only when the task needs one
- Tunes the local runtime from the UI: GPU offload layers, context size, KV-cache type (f16/q8_0/q4_0 and more), parallel slots, and multi-GPU splitting
- Keeps prompts and data on your device for local runs, so private or regulated content never leaves the machine
- Sits in the same workspace as your BYOK cloud keys — chat, agents, apps, code, and knowledge all draw from local or cloud without switching tools
Use cases
- Privacy-conscious engineer: Runs a local GGUF model so contract and codebase prompts never leave the workstation, and lets Maestro escalate to a cloud key only for the occasional harder query
- Solo builder on a budget: Leans on local inference for everyday chat and agent work to keep per-token cloud spend near zero, tuning GPU offload and context size to fit a single laptop
- Workstation power user: Splits a large model across multiple GPUs and raises parallel slots to serve several agents at once, entirely on-device
Frequently asked questions
Do I need an API key to run a model locally?
No. Local runs use a model file on your own machine and a built-in inference server, so nothing is billed per token and no key is required. You only need a key when Maestro routes a request to a cloud model, and that key is your own — osFoundry doesn't resell access or add a markup.
How does osFoundry decide between local and cloud?
Maestro handles routing automatically. It prefers the local server whenever the request fits what your machine can run, and falls back to a cloud model only when the task needs one. You configure the local runtime (GPU layers, context size, cache type) but you don't have to route each message by hand.
What kind of models and hardware does the local server support?
It loads GGUF-format models from a folder on your machine and serves them through a llama.cpp-style runtime. You can offload layers to a GPU, adjust the context window and KV-cache quantization to fit your memory, run multiple parallel request slots, and split across multiple GPUs. Exactly which models run well depends on your RAM, VRAM, and CPU.