Can a Mac run large language models locally without a dedicated GPU?

Yes — that's Apple Silicon's strength. Its unified memory is shared between the CPU and GPU and is high-bandwidth, so the GPU can use the whole memory pool. A 64GB Mac can run a 70B model at 4-bit, which a 24GB discrete GPU can't fit, and smaller models run fast. No separate GPU is needed.

How much unified memory do I need to run a 7B, 30B, or 70B model?

At 4-bit quantization: about 16GB for a 7B–8B model, 32GB for up to roughly 30B, and 64GB for a 70B (which is around 40GB of weights, leaving tight context headroom). 128GB lets you run a 70B at higher precision with room for large context. Always leave headroom for the KV cache, which grows with context length.

Is MLX faster than llama.cpp on Apple Silicon?

Often, on smaller models — MLX is Apple's framework built for unified memory and is meaningfully faster below around 14B, and it shines on mixture-of-experts models. Above roughly 27B the two converge because both become bandwidth-bound. llama.cpp's advantage is the larger GGUF model library and cross-platform consistency. Use MLX for peak speed on smaller and MoE models.

Why is my Mac slow to start responding on long prompts?

Because prompt processing (time to first token) is Apple Silicon's weak spot. Prefill is compute-bound and Apple's GPUs lack the dedicated tensor cores NVIDIA uses for it, so a long prompt takes a while before the first token appears — even when token generation afterward is fast. It's the main Apple-specific limitation for long-context use.

What's the fastest tool to run LLMs on a Mac?

For peak performance, MLX-LM (Apple's framework), especially on smaller and MoE models. For ease of use, Ollama (CLI plus an OpenAI-compatible API) or LM Studio (the best GUI) are the popular choices, and several now use MLX backends on Apple Silicon. The right pick depends on whether you want maximum speed or maximum convenience.

Can I run a 70B model on a 64GB MacBook Pro?

Yes, at 4-bit — a 70B at Q4 is around 40GB, which fits in 64GB with room for the runtime and a modest context window. It's the practical entry point for 70B on a Mac; context headroom is limited, so for large contexts or higher precision you'd want 128GB. Expect solid generation speed but slower time-to-first-token on long prompts.

Apple Silicon or NVIDIA for local LLMs?

Apple Silicon is excellent for private, single-user local inference — large unified memory lets one Mac run models that need multiple discrete GPUs, at lower power. NVIDIA wins on raw bandwidth (several times higher), prompt-processing speed, high-concurrency serving, and ecosystem (CUDA). Choose Apple for personal local use and big models on one machine; NVIDIA for throughput and long-context serving.

← Resources

By Ren Sugaya — Performance Engineer, GPU & Inference · 2026-05-02 · TUTORIAL

Running LLMs on Apple Silicon: Unified Memory and What Fits

Apple Silicon is unusually good at running LLMs because its unified memory is shared, high-bandwidth, and large — a 64GB Mac can run a 70B model. This guide covers what fits on each RAM tier, MLX versus llama.cpp, the one real weakness (prompt processing), and the tooling.

Why unified memory is the advantage

Apple Silicon shares a single high-bandwidth memory pool across the CPU, GPU, and Neural Engine — there's no separate VRAM and no slow copy across a PCIe bus. That's why a 64GB Mac can load a model that won't fit on a 24GB discrete GPU: the whole pool is available to the GPU.

It matters because LLM token generation is [memory-bandwidth-bound](/articles/running-llms-on-cpu) — each token requires reading the model from memory — and Apple's unified bandwidth is high enough to be genuinely competitive on decode. Bandwidth, not raw compute, is the spec that decides generation speed here.

What fits, by memory tier

Unified memory is the binding constraint, so size the model to your Mac (figures at 4-bit quantization):

16GB: 7B–8B models comfortably.
32GB: up to roughly 30B models.
64GB: a 70B model becomes feasible (a 70B at Q4 is around 40GB), though context headroom is tight — this is the practical entry point for 70B.
128GB: a 70B at higher precision (Q5/Q8) with room for large context.
Ultra-tier (192GB+): 70B at high precision, very large models, or several models resident at once.

Rule of thumb: budget roughly 0.5–0.6 GB per billion parameters for the weights at Q4, then leave room for the KV cache, which grows with context length and is often what decides whether you fit 16K or 32K context.

MLX vs llama.cpp

Two runtimes dominate on Mac. llama.cpp's Metal backend translates GPU compute into Metal shaders and uses the cross-platform GGUF format with the largest model library. MLX is Apple's own framework, built ground-up for unified memory — generally faster on smaller models (often meaningfully so under ~14B), with the gap narrowing above ~27B where both hit the bandwidth ceiling.

MLX has momentum: Apple positioned it as the preferred Apple Silicon framework, popular runtimes added MLX backends, and it shines on mixture-of-experts models. The pragmatic take: MLX for peak speed on smaller and MoE models, llama.cpp/GGUF for the widest model selection and cross-platform consistency.

The real weakness: prompt processing

Apple Silicon's honest limitation isn't generation — it's prompt processing (time to first token). Prefill is compute-bound, involving large matrix multiplications, and Apple's GPUs lack the dedicated tensor cores that make NVIDIA fast here. So even when token generation is competitive, the model can be noticeably slow to start responding on a long prompt, which dominates the real-world feel of long-context work.

The other limitation is ecosystem: most tooling targets CUDA, there are fewer pre-quantized MLX models than GGUF, and a high-end NVIDIA GPU still has several times the raw bandwidth. For interactive single-user local use, Apple Silicon is excellent; for long-context or high-throughput serving, those gaps matter.

Tooling and quantization

The Mac tooling landscape is mature: Ollama (CLI plus an OpenAI-compatible API, the easiest start), LM Studio (the best GUI), MLX-LM (peak performance and fine-tuning, via Python), and privacy-focused offline apps. GGUF is the dominant cross-tool quantization format; the MLX format is Apple-native and faster but has a smaller library.

As everywhere, [quantization](/articles/llm-quantization-explained-gguf) is what makes it work — dropping to 4-bit shrinks a model roughly fourfold (a 70B from ~140GB to ~40GB) with modest quality loss, and Q4_K_M is the usual sweet spot. Step up to Q5 or Q8 if your unified memory allows.

Apple Silicon in osFoundry

osFoundry runs local inference on Apple Silicon through the Metal and MLX paths, so the model executes entirely on your Mac's GPU using its unified memory — no data leaves the device and there's no PCIe copy penalty. Because the constraint is simply how much unified memory you have, osFoundry helps you pick a model sized to your Mac — a 7B–8B model on a 16GB machine, up to a 70B on a 64GB+ machine — and selects the quantization to fit the model into available memory while preserving as much quality as possible, so you don't reason about GGUF formats or Q4/Q5/Q8 trade-offs by hand. The result is fully private, offline-capable local inference tuned to each Mac's memory tier — and the same model can scale out to a dedicated GPU endpoint when you need faster prompt processing or higher throughput. (For the underlying bandwidth story, see our [CPU inference guide](/articles/running-llms-on-cpu).)

Frequently asked questions

Can a Mac run large language models locally without a dedicated GPU?: Yes — that's Apple Silicon's strength. Its unified memory is shared between the CPU and GPU and is high-bandwidth, so the GPU can use the whole memory pool. A 64GB Mac can run a 70B model at 4-bit, which a 24GB discrete GPU can't fit, and smaller models run fast. No separate GPU is needed.
How much unified memory do I need to run a 7B, 30B, or 70B model?: At 4-bit quantization: about 16GB for a 7B–8B model, 32GB for up to roughly 30B, and 64GB for a 70B (which is around 40GB of weights, leaving tight context headroom). 128GB lets you run a 70B at higher precision with room for large context. Always leave headroom for the KV cache, which grows with context length.
Is MLX faster than llama.cpp on Apple Silicon?: Often, on smaller models — MLX is Apple's framework built for unified memory and is meaningfully faster below around 14B, and it shines on mixture-of-experts models. Above roughly 27B the two converge because both become bandwidth-bound. llama.cpp's advantage is the larger GGUF model library and cross-platform consistency. Use MLX for peak speed on smaller and MoE models.
Why is my Mac slow to start responding on long prompts?: Because prompt processing (time to first token) is Apple Silicon's weak spot. Prefill is compute-bound and Apple's GPUs lack the dedicated tensor cores NVIDIA uses for it, so a long prompt takes a while before the first token appears — even when token generation afterward is fast. It's the main Apple-specific limitation for long-context use.
What's the fastest tool to run LLMs on a Mac?: For peak performance, MLX-LM (Apple's framework), especially on smaller and MoE models. For ease of use, Ollama (CLI plus an OpenAI-compatible API) or LM Studio (the best GUI) are the popular choices, and several now use MLX backends on Apple Silicon. The right pick depends on whether you want maximum speed or maximum convenience.
Can I run a 70B model on a 64GB MacBook Pro?: Yes, at 4-bit — a 70B at Q4 is around 40GB, which fits in 64GB with room for the runtime and a modest context window. It's the practical entry point for 70B on a Mac; context headroom is limited, so for large contexts or higher precision you'd want 128GB. Expect solid generation speed but slower time-to-first-token on long prompts.
Apple Silicon or NVIDIA for local LLMs?: Apple Silicon is excellent for private, single-user local inference — large unified memory lets one Mac run models that need multiple discrete GPUs, at lower power. NVIDIA wins on raw bandwidth (several times higher), prompt-processing speed, high-concurrency serving, and ecosystem (CUDA). Choose Apple for personal local use and big models on one machine; NVIDIA for throughput and long-context serving.