Local model guide · updated July 2026 · 10 min read

How to Run Mistral Locally

Mistral's open line is the pragmatic European choice: permissive licenses, lean dense models, and a coding-specialist worth its own mention. The 24B Mistral Small runs on a mainstream GPU, Devstral targets agentic coding on the same hardware, and Mistral Large is the multi-GPU option. This guide runs each across Ollama, llama.cpp, and vLLM, with sampler notes and tool calling.

Which Mistral

For most people Mistral Small is the answer: a capable 24B general model on Apache-2.0 that fits a single card. Reach for Devstral when the workload is code, and Large only when you have the hardware for a 600B-class model.

Run with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama run mistral-small
# coding specialist:
ollama run devstral

Run with llama.cpp

Build llama.cpp (steps here), then serve Mistral Small:

./llama.cpp/llama-server \
  -hf unsloth/Mistral-Small-3.2-24B-Instruct-2506-GGUF:Q4_K_M \
  --jinja \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8080

Devstral for coding agents

Devstral is tuned for software engineering and agentic edit-run-fix loops, and it is about the same size as Mistral Small, so it runs on the same hardware. Run it the same way, then point a coding agent at its OpenAI-compatible endpoint by setting the agent's base URL to your local server. Because Devstral is trained for tool use, it drives an agent's file and shell tools well once the endpoint is wired up.

./llama.cpp/llama-server \
  -hf unsloth/Devstral-Small-2505-GGUF:Q4_K_M \
  --jinja -ngl 99 -c 32768 --host 127.0.0.1 --port 8080

Serve with vLLM

vllm serve mistralai/Mistral-Small-3.2-24B-Instruct-2506 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser mistral

Add --tensor-parallel-size N for Mistral Large across several GPUs.

On a Mac

pip install mlx-lm
mlx_lm.server --model mlx-community/Mistral-Small-3.2-24B-4bit --port 8080

LM Studio lists the Mistral models with one-click download and a local server.

Sampler settings

Mistral models tend to prefer a lower temperature than most, and the current Small checkpoint recommends a notably low temperature, around 0.15, for its best behavior. Confirm the exact value on the model card, since it varies by checkpoint, but as a rule start lower with Mistral than you would with a Qwen or Gemma model, and raise only if answers feel too rigid.

Tool calling

Mistral has first-class tool calling. In vLLM use --tool-call-parser mistral with --enable-auto-tool-choice. In llama.cpp pass --jinja so the template formats tools. The endpoint is OpenAI-compatible, which is why Devstral plus a coding agent is a clean local pairing.

Troubleshooting

• Rambling or off-task output. Lower the temperature; Mistral prefers it lower than other families.
• Tool calls not parsed. Missing --jinja in llama.cpp, or the wrong parser in vLLM. Use mistral.
• Out of memory. Drop a quant tier; Small at Q8 needs more than a 16GB card. Large needs multi-GPU or offload.
• Truncated prompts. Raise Ollama's num_ctx or llama.cpp's -c.

FAQ

What GPU runs Mistral Small? A 24B dense model at 4-bit is about 14 to 15 GB, so a 16GB GPU is comfortable and a 24GB card is easy.

What is Devstral? Mistral's coding specialist, about the same size as Small, tuned for software engineering and agentic coding. Pair it with a coding agent over its OpenAI endpoint.

Tool calling? Yes. Use --tool-call-parser mistral in vLLM or --jinja in llama.cpp.