Run open-weight models on your own hardware

Engine-by-engine guides for running local LLMs, deeper than the model cards. Each covers Ollama, llama.cpp, vLLM, LM Studio, and MLX on Apple Silicon, with real VRAM math, quant selection, tool calling, an OpenAI-compatible endpoint, and the failure modes nobody documents.

Model guides

Kimi K3 Moonshot · trillion-scale MoE The frontier open MoE. Dynamic quants, MoE-to-CPU offload, and multi-GPU serving for a model that does not fit one card. Qwen3 Alibaba · 4B to 480B, dense + MoE The everyman family. Runs on a 4090 at 32B, scales to 480B Coder. Thinking mode, tool calling, and the sweet-spot quants. gpt-ossOpenAI · 20B / 120B · Apache-2.0Native MXFP4 makes the 120B fit one GPU. Harmony format, reasoning_effort, and every engine. DeepSeek V3.2 / R1DeepSeek · 671B MoE · MITDistills on a consumer GPU, the full 671B on a big-RAM box with dynamic quants and expert offload. Gemma 3Google · 1B to 27B dense, visionThe phone-to-workstation dense line. The 27B on a 4090, image input from 4B up. MistralMistral · Small / Large 3 · DevstralThe 24B Small on a mainstream GPU, Devstral for coding agents, Large for multi-GPU. Llama 4Meta · Scout / Maverick MoENative multimodal, huge context. Scout on a 64GB Mac or 24GB card with offload. GLM 4.6 / 5z.ai · MoE · MITStrong agentic coder. Air on a single card with offload, wired straight into a coding agent.

Read this first

Every model guide assumes the same four decisions, so learn them once. Which engine: Ollama for the fastest start, llama.cpp for control and huge-model offload, vLLM for multi-GPU throughput and an OpenAI-compatible server, LM Studio or MLX on a Mac. Which quant: Q4_K_M is the reliable 4-bit default, Q8_0 is near-lossless, and dynamic low-bit quants are what make trillion-parameter models fit at all.

How much memory: weight bytes are roughly params times bits-per-weight divided by eight, so a 32B model at 4-bit is about 18 to 20 GB, plus headroom and KV cache. A mixture-of-experts model needs memory for its total parameters but runs at the speed of its active parameters, which is why a 1T model with 32B active can run on a big-RAM box at usable speed. How you use it: every engine here can expose an OpenAI-compatible endpoint, so any agent or SDK drives a local model by pointing its base URL at your machine.

For the full breakdown of what fits a Mac Mini, MacBook Pro, Mac Studio, or a given GPU, and how much RAM each model needs, see the hardware and RAM guide.