Local model guide · updated July 2026 · 11 min read

How to Run gpt-oss Locally

gpt-oss is OpenAI's open-weight release, and it is one of the friendliest large models to run at home because it ships pre-quantized in native MXFP4. The 20B fits a laptop-class machine and the 120B fits a single data-center GPU. The one thing to get right is the harmony response format. This guide covers both sizes across Ollama, llama.cpp, vLLM, and LM Studio, plus reasoning effort, tool calling, and the failure modes.

20B or 120B

Both are mixture-of-experts models with very small active-parameter counts, so they run fast for their size. Pick by memory.

The 20B is the everyday choice: a strong reasoner that runs on a mainstream laptop GPU. The 120B is close to frontier quality and still fits one 80GB card, which is unusual for a model this capable.

Why MXFP4 matters

gpt-oss was trained and released in MXFP4, a 4-bit floating-point format for the expert layers. That means the download is already small and there is no separate quantization step to reason about: the 4-bit weights are the real weights, not a lossy compression of a larger checkpoint. You can still find higher-precision GGUF conversions, but the native MXFP4 is what makes the 120B fit a single GPU, so prefer it unless you have a specific reason not to.

Run with Ollama

Ollama is the fastest path and handles the harmony template for you:

curl -fsSL https://ollama.com/install.sh | sh
ollama run gpt-oss:20b
# or the big one:
ollama run gpt-oss:120b

Raise the context window with a Modelfile if you feed long prompts, since the default is low:

printf 'FROM gpt-oss:20b\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create gpt-oss-32k -f Modelfile

Run with llama.cpp

Build llama.cpp (see the build steps), then serve the GGUF with the model's template so harmony formatting is correct:

./llama.cpp/llama-server \
  -hf unsloth/gpt-oss-20b-GGUF:F16 \
  --jinja \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8080

There is also a Docker path if you prefer a container: docker model run hf.co/unsloth/gpt-oss-20b-GGUF:F16.

Serve with vLLM

For throughput and a production OpenAI-compatible server, use vLLM with the openai tool parser:

vllm serve openai/gpt-oss-120b \
  --tensor-parallel-size 2 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser openai

Set --tensor-parallel-size to your GPU count for the 120B, or drop to 1 for the 20B on a single card.

On a Mac

The 20B runs comfortably on Apple Silicon; the 120B wants 64GB or more of unified memory. Use MLX or LM Studio:

pip install mlx-lm
mlx_lm.server --model mlx-community/gpt-oss-20b --port 8080

LM Studio lists both sizes, downloads the right build, and gives you a local server at http://localhost:1234/v1.

Harmony and reasoning effort

gpt-oss emits a structured format called harmony, which separates its output into an analysis channel (the reasoning), a commentary channel (where tool calls live), and a final channel (the answer). You do not usually assemble this by hand: Ollama, llama.cpp with --jinja, and vLLM apply the harmony template. What you do control is how hard the model thinks, through a reasoning-effort setting of low, medium, or high. Higher effort means longer analysis and better answers on hard problems, at the cost of more tokens. The recommended sampler is a temperature of 1.0 with top-p 1.0, which is unusual but correct for this model, so do not lower them the way you would for other models.

Tool calling

Tool calls flow through the commentary channel of the harmony format. In vLLM, --tool-call-parser openai extracts them into standard OpenAI tool-call objects. In llama.cpp, --jinja plus the model's template does the same. Because the endpoint speaks the OpenAI API, an agent framework passes its tools array unchanged and reads the calls back normally.

Troubleshooting

• Malformed or empty answers. The harmony template is not being applied. Use Ollama, or pass --jinja in llama.cpp, or the openai parser in vLLM. Do not hand-write the prompt.
• Output looks like raw channel markers. Your client is not parsing the final channel. Use an engine that understands harmony, or the OpenAI endpoint rather than raw completion.
• Out of memory on 120B. Use the native MXFP4 build, add a second GPU with tensor parallel, or move to a larger Mac; do not convert it up to FP16.
• Weak reasoning. Raise reasoning effort to high, and keep temperature at 1.0 rather than lowering it.

FAQ

What hardware runs the 20B? Native MXFP4 makes its weights about 12 to 14 GB, so a 16GB GPU or Mac runs it comfortably, and an 8GB card works with offload. The 120B needs roughly 65 GB, so an 80GB GPU or a 64GB-plus Mac.

What is harmony? The structured response format gpt-oss was trained on, with analysis, commentary, and final channels. Use the model's template or output and tool calls will not parse.

Does it do tool calling? Yes, via the commentary channel. Use --tool-call-parser openai in vLLM or --jinja in llama.cpp; the endpoint is OpenAI-compatible.