osFoundry / Platform
Benchmarks
Side-by-side scores across MMLU, GSM8K, HumanEval and others. Pick a model with data.
What it is
Benchmarks is the model-comparison view in osFoundry, the bring-your-own-key desktop AI workspace. It puts published benchmark scores for the models you can run next to each other in one table, so instead of guessing from a vendor's headline number you see how models line up on the tasks that map to your work: MMLU for broad knowledge and reasoning, GSM8K for grade-school math and multi-step arithmetic, HumanEval for code generation, and other standard suites alongside them.
Because osFoundry runs any model — local, cloud, or self-hosted — from the same workspace, the comparison is not limited to one provider's catalog. You can weigh a hosted frontier model against a smaller open-weight model you run locally, read the scores in the same columns, and pick the one the data actually supports for your use case. Once you decide, that model is already available to use with your own API key: no reseller markup on top of the provider's price, and nothing you evaluate here has to leave your device to be compared.
The point is a defensible choice, not a leaderboard for its own sake. A model that tops MMLU may trail on HumanEval; a local model may score close enough on GSM8K that running it privately and at no per-seat cost is the better call. Benchmarks makes that trade-off visible before you commit a model to a chat, an agent, or a pipeline, so model selection in osFoundry is grounded in numbers rather than reputation.
Key capabilities
- Side-by-side scores across MMLU, GSM8K, HumanEval and other standard suites in one comparison table
- Covers models you can actually run in osFoundry — local, cloud, and self-hosted — not a single provider's catalog
- Match the benchmark to the job: knowledge and reasoning (MMLU), math (GSM8K), code (HumanEval)
- Decide, then run the model directly with your own API key — no reseller markup added to the provider's price
- Compare open-weight local models against hosted frontier models to trade off accuracy, privacy, and cost
Use cases
- Developer: compare HumanEval scores across candidate models before wiring one into a coding agent
- Team lead: check whether a local open-weight model scores close enough on MMLU and GSM8K to run privately with no per-seat fees
- Researcher: line up several models' published scores in one view to pick a defensible default for a workspace
Frequently asked questions
Which benchmarks can I compare models on?
Standard published suites shown side by side, including MMLU (broad knowledge and reasoning), GSM8K (grade-school and multi-step math), and HumanEval (code generation), with other suites listed alongside them so you can match the metric to your task.
Does osFoundry run the benchmarks itself?
The view compares published benchmark scores for the models you can run, so you can pick a model backed by data. It is a model-selection aid: the goal is a grounded choice before you commit a model to a chat, agent, or pipeline.
After I pick a model from the benchmarks, how do I use it?
The models you compare are the ones osFoundry can run — local, cloud, or self-hosted. Once you choose, you use it with your own API key at the provider's price with no reseller markup, and local models can stay entirely on your device.