Local model guide · updated July 2026 · 13 min read

How to Run Qwen3 Locally

Qwen3 is the family most people should start with, because it spans a laptop-sized 4B up to a 480B coding model and it runs cleanly on every local engine. This guide matches a variant to your hardware, then walks Ollama, llama.cpp, vLLM, and Apple Silicon, with the sampler settings the Qwen team recommends, how to toggle thinking mode, tool calling, long context with YaRN, and an OpenAI-compatible endpoint for your agents.

Pick a size for your hardware

Qwen3's range is its strength. Choose the largest variant whose 4-bit weights leave room for context on your device. Weight size is roughly parameters times bits-per-weight over eight, so a 32B model at 4-bit is about eighteen to twenty gigabytes, which is why it lands squarely on a 24GB card.

Note the mixture-of-experts variants. Qwen3-Next-80B and Qwen3-235B activate only a fraction of their parameters per token, so they compute like much smaller models but still need memory for the full parameter count. On a big-RAM box or a high-memory Mac they punch far above what their size suggests.

Quant selection

For dense Qwen3 the choice is simple. Q4_K_M is the reliable default and where most people live; Q5_K_M or Q6_K if you have headroom and want a touch more fidelity; Q8_0 when memory is plentiful and you want near-lossless. Below 4-bit, quality falls off faster on the smaller dense models, so avoid 2-bit unless you are running one of the very large MoE variants, where dynamic low-bit quants keep quality high because only the less-sensitive expert layers are pushed low.

Fastest start: Ollama

For any Qwen3 up to 32B, Ollama is the quickest path. Install it, then run a tag from the library or a GGUF straight from Hugging Face:

curl -fsSL https://ollama.com/install.sh | sh
ollama run qwen3:32b
# or any GGUF from Hugging Face, choosing a quant:
ollama run hf.co/unsloth/Qwen3-32B-GGUF:Q4_K_M

One gotcha dominates Ollama support threads: the default context window is only 2048 tokens, so long prompts silently truncate. Raise it with a Modelfile:

printf 'FROM qwen3:32b\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create qwen3-32k -f Modelfile
ollama run qwen3-32k

Control: llama.cpp

When you want explicit control over offload, context, and KV cache, use llama.cpp. Build it once (see the build steps in the Kimi K3 guide), then start the server:

./llama.cpp/llama-server \
  -hf unsloth/Qwen3-32B-GGUF:Q4_K_M \
  --jinja \
  -ngl 99 \
  -c 32768 \
  -fa on \
  --host 127.0.0.1 --port 8080

-ngl 99 puts all layers on the GPU (drop it if you need to offload some to CPU on a smaller card), -fa on enables flash attention to save KV-cache memory, and --jinja uses Qwen3's chat template so thinking tags and tool calls format correctly.

Throughput: vLLM

For serving multiple users or maximizing tokens per second on a capable GPU, vLLM is the tool, and it gives you an OpenAI-compatible server out of the box:

vllm serve Qwen/Qwen3-32B \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90 \
  --enable-auto-tool-choice \
  --tool-call-parser hermes

Add --tensor-parallel-size N to shard across N GPUs for the larger variants, and --quantization awq or gptq if you are loading a pre-quantized GPU checkpoint rather than full precision.

On a Mac: MLX and LM Studio

On Apple Silicon, MLX is the native, fastest path and uses unified memory well:

pip install mlx-lm
mlx_lm.generate --model mlx-community/Qwen3-32B-4bit --prompt "Write a haiku about GPUs."
mlx_lm.server --model mlx-community/Qwen3-32B-4bit --port 8080

LM Studio is the no-terminal option: search Qwen3 in the app, download a quant, and flip on the local server at http://localhost:1234/v1. It picks the GGUF or MLX runtime for you.

The large MoE variants

Qwen3-Next-80B, Qwen3-235B, and Qwen3-Coder-480B are mixture-of-experts models. On a single big GPU plus system RAM, run them in llama.cpp with expert offload, exactly like the large-model path: add -ot ".ffn_.*_exps.=CPU" to keep the experts in RAM and a dynamic quant to fit. On a multi-GPU server, prefer vLLM with --tensor-parallel-size. On a 128GB or larger Mac, MLX at 4-bit holds Next-80B or 235B comfortably.

Serve an OpenAI-compatible API

Whichever engine you pick exposes /v1/chat/completions. Point any OpenAI client at the local base URL and use any string for the key:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3","messages":[{"role":"user","content":"one-line hello"}]}'

Tool calling

Qwen3 is a strong tool caller. In llama.cpp, --jinja is what makes tool formatting correct. In vLLM, pass --enable-auto-tool-choice and --tool-call-parser hermes (Qwen3-Coder uses its own parser, so check the model card). Send an OpenAI-style tools array and read the tool call from the response. Because it is the OpenAI shape, coding agents and frameworks that accept a custom base URL drive it without changes.

Thinking mode and samplers

Qwen3 has an explicit reasoning mode, and the correct sampler differs by mode. Toggle it inline in llama.cpp and Ollama by appending /think or /no_think to your message, or in transformers and vLLM by setting enable_thinking in the chat template call.

The one rule to remember: never greedy-decode in thinking mode. A temperature of zero makes the reasoning loop or repeat. Keep temperature at the values above.

Long context with YaRN

Qwen3's native window is around 32K, and it extends to 131K or beyond on the large variants using YaRN, a context-scaling method the model enables through config. Two cautions. First, extended context multiplies KV-cache memory, so only raise -c or --max-model-len to what you need and quantize the cache with --cache-type-k q8_0 if it is tight. Second, enabling YaRN scaling can slightly reduce quality on short prompts, so turn it on only when you actually feed long inputs.

Troubleshooting

• Truncated long prompts. Ollama's default context is 2048. Set num_ctx in a Modelfile, or -c in llama.cpp.
• Reasoning loops or repetition. You are likely greedy-decoding in thinking mode. Use temp 0.6, top-p 0.95.
• Gibberish. Wrong chat template. Pass --jinja in llama.cpp; confirm you pulled the instruct model.
• Out of memory. Drop a quant tier, lower context, offload experts with -ot on the MoE variants, or reduce --gpu-memory-utilization in vLLM.
• Tool calls not parsed. Missing --jinja (llama.cpp) or the wrong --tool-call-parser (vLLM).

FAQ

What GPU runs Qwen3 32B? A 24GB card such as a 4090 or 3090 runs 32B at 4-bit comfortably. 16GB runs 14B, 8GB runs 4B. Higher precision or the large MoE variants need more VRAM or system RAM with offload.

How do I toggle thinking mode? Append /think or /no_think in llama.cpp and Ollama, or set enable_thinking in the chat template for transformers and vLLM. Use the thinking sampler when it is on.

Does it do tool calling? Yes. Pass --jinja in llama.cpp, or --enable-auto-tool-choice --tool-call-parser hermes in vLLM. The endpoint is OpenAI-compatible.