Local model guide · updated July 2026 · 12 min read

How to Run DeepSeek R1 and V3.2 Locally

DeepSeek's flagship reasoners are 671B-class mixture-of-experts models, which sounds out of reach for a home machine, and mostly is at full size. But there are two honest local paths: the distilled models, which put much of R1's reasoning into dense Qwen and Llama checkpoints that run on a normal GPU, and the full model, which runs on a big-RAM box with dynamic quants and expert offload. This guide covers both.

Full or distilled

Decide this first, because the two paths have almost nothing in common.

The distills are the practical answer for most people: they capture a large share of the reasoning behavior and run like any dense model of their size. The full model is worth the effort only if you have the memory and want the top of the quality curve.

The distills on a GPU

The distills run in one command with Ollama. Raise the context, since these are reasoning models that produce long thought traces:

curl -fsSL https://ollama.com/install.sh | sh
ollama run deepseek-r1:32b
# smaller options: deepseek-r1:8b, deepseek-r1:14b, deepseek-r1:70b
printf 'FROM deepseek-r1:32b\nPARAMETER num_ctx 32768\n' > Modelfile
ollama create r1-32k -f Modelfile

In llama.cpp or vLLM you can point at the Hugging Face distill repos directly, for example deepseek-ai/DeepSeek-R1-Distill-Qwen-32B.

Dynamic quants for the full model

The full 671B does not fit any single GPU, so quantization is not optional, it is the whole strategy. Dynamic low-bit quants keep the sensitive attention layers at higher precision and push the mixture-of-experts layers down to one or two bits, so the model stays close to full quality while shrinking dramatically. Approximate sizes for the full model give you the memory target to hit:

The full 671B with offload

Run the full model in llama.cpp with the experts offloaded to system RAM. Match the quant to your combined VRAM plus RAM:

./llama.cpp/llama-server \
  -hf unsloth/DeepSeek-R1-GGUF:IQ1_S \
  --jinja \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  --cache-type-k q4_0 \
  --temp 0.6 --top-p 0.95 --min-p 0.01 \
  -c 16384 --host 127.0.0.1 --port 8080

The -ot ".ffn_.*_exps.=CPU" flag is what makes this possible, keeping the expert layers in RAM while attention stays on the GPU. Expect a few tokens per second on a 24GB card paired with 128 to 256 GB of RAM; throughput is bounded by memory bandwidth, not the GPU. If the quant ships as multiple shards, merge them with llama-gguf-split --merge.

Multi-GPU with vLLM

On a server with several data-center GPUs, vLLM serves the full model at production speed and gives you an OpenAI-compatible endpoint:

vllm serve deepseek-ai/DeepSeek-V3.2-Exp \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --enable-auto-tool-choice \
  --tool-call-parser deepseek_v3

Reasoning and sampler

R1 and V3.2 are reasoning models that emit a thinking trace before the answer, so give them room in the context and use the recommended sampler: a temperature around 0.6 with top-p 0.95, and a small min-p of about 0.01. Do not greedy-decode; a temperature of zero makes the reasoning loop. If you only want the final answer, strip the thinking section on the client side rather than suppressing it, since the model reasons better when allowed to think.

Tool calling

Enable tools with the DeepSeek parser. In vLLM use --tool-call-parser deepseek_v3 (or the matching version parser for the exact checkpoint) together with --enable-auto-tool-choice. In llama.cpp, --jinja applies the model's tool template. The endpoint is OpenAI-compatible, so an agent passes its tool schemas unchanged.

Serve an API

Every path above exposes /v1/chat/completions. Point an OpenAI client at the local base URL with any key:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Prove that sqrt(2) is irrational."}]}'

Troubleshooting

• Reasoning loops or repeats. You are greedy-decoding. Use temp 0.6, top-p 0.95, min-p 0.01.
• Full model painfully slow. Expected on offload; a few tokens per second is normal. Use a smaller quant, add RAM, or move to multi-GPU for speed.
• Out of memory on the full model. Drop to a smaller dynamic quant, offload more experts, quantize the KV cache, and lower context.
• Distill gives shallow answers. Raise context so the thinking trace is not truncated, and keep the reasoning sampler.
• Multi-part GGUF will not load. Merge with llama-gguf-split --merge, or point at the first shard in llama.cpp.

FAQ

Can I run R1 on a consumer GPU? Not the full 671B, but the distills yes. The 32B distill runs on a 24GB card at 4-bit. The full model needs a big-RAM box with offload or multi-GPU.

How much RAM for the full model? A dynamic quant is roughly 160 to 250 GB. A 24GB GPU plus 128 to 256 GB of RAM runs it with expert offload at a few tokens per second.

Sampler settings? Temperature around 0.6, top-p 0.95, min-p about 0.01. Never greedy-decode a reasoning model.