Local model guide · updated July 2026 · 14 min read

How to Run Kimi K3 Locally

Kimi K3 is Moonshot's frontier open mixture-of-experts model, and running it at home is less about a fast GPU than about memory strategy. This guide walks the whole path: sizing your hardware honestly, choosing a quant that fits, offloading the expert layers to system RAM in llama.cpp, serving it across GPUs with vLLM, running it on a big Apple Silicon machine, wiring an OpenAI-compatible endpoint to your agent, and fixing the things that break.

Can you run it?

The first thing to understand about a mixture-of-experts model is that memory and speed are decoupled. The model routes each token through a small slice of its parameters, the active experts, so it computes at the speed of a much smaller model. But it still has to hold every expert in memory, because any token might route to any of them. So Kimi K3 needs memory sized to its total parameter count and runs at the speed of its active count. That single fact is why a trillion-parameter model is runnable at home at all: you pair cheap, plentiful RAM with a modest GPU, and the throughput stays usable.

Estimate the weight size directly. Bytes are roughly parameters times bits-per-weight divided by eight. A 4-bit quant is about 4.5 to 5 bits-per-weight, so a trillion-parameter model at 4-bit is on the order of hundreds of gigabytes, and a dynamic low-bit quant brings that down further. Add ten to twenty percent headroom plus the KV cache, which grows with context length. Match your combined VRAM plus system RAM to the quant file size and you get roughly six or more tokens per second; fall short and it still runs, spilling to SSD, just slowly.

Pick a quant

Quantization trades bits per weight for size. The reliable tiers, smallest to largest: aggressive dynamic low-bit quants for fitting the model at all, Q4_K_M as the standard 4-bit default, Q5_K_M and Q6_K for higher fidelity, and Q8_0 as a near-lossless baseline. For a model this large the decision is usually made for you by memory: you take the largest quant whose file fits your VRAM plus RAM.

The reason a trillion-parameter model fits on a workstation is dynamic quantization. Instead of quantizing every weight to the same width, it keeps the attention and other sensitive layers at higher precision and pushes the mixture-of-experts layers, which dominate the parameter count, down to one or two bits. The result is a file close to a uniform low-bit quant in size but far closer to full precision in quality. These are the quants that make the difference between "cannot load" and "runs tonight."

Run with llama.cpp

For a model of this size, llama.cpp is the primary path. Ollama is excellent but is aimed at models up to roughly 32B and a single GGUF file; Kimi K3 needs expert offload and often ships as multi-part shards, which llama.cpp handles directly. Build it once:

git clone https://github.com/ggml-org/llama.cpp
cmake llama.cpp -B llama.cpp/build \
  -DBUILD_SHARED_LIBS=OFF -DGGML_CUDA=ON -DLLAMA_CURL=ON
cmake --build llama.cpp/build --config Release -j --clean-first \
  --target llama-cli llama-server llama-gguf-split
cp llama.cpp/build/bin/llama-* llama.cpp/

Set -DGGML_CUDA=OFF on a CPU-only or Apple machine; Metal is on by default on macOS. Then pull a quant straight from Hugging Face and start the server. Replace the repo id with the real one from the model card.

./llama.cpp/llama-server \
  -hf unsloth/Kimi-K3-GGUF:UD-Q2_K_XL \
  --jinja \
  -ngl 99 \
  -c 16384 \
  --host 127.0.0.1 --port 8080

-ngl 99 offloads all layers it can to the GPU, -c sets context, and --jinja tells the server to use the model's own chat template, which you want for correct formatting and tool calls. If the quant shipped as multiple shards, point at the first shard, or merge them first:

./llama.cpp/llama-gguf-split --merge \
  Kimi-K3-UD-Q2_K_XL-00001-of-000NN.gguf  kimi-k3-merged.gguf

MoE offload for big models

This is the move that makes a trillion-parameter model run on one GPU. The -ot flag matches tensor names with a regex and pins them to a device. Send the expert feed-forward tensors to the CPU so they live in system RAM, and keep attention on the GPU:

./llama.cpp/llama-server \
  -hf unsloth/Kimi-K3-GGUF:UD-Q2_K_XL --jinja \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  --cache-type-k q4_0 \
  -c 16384 --host 127.0.0.1 --port 8080

Start by offloading all experts with -ot ".ffn_.*_exps.=CPU". If you have VRAM to spare and want more speed, offload fewer of them, for example only the up and down projections, and let the gate stay on GPU: -ot ".ffn_(up|down)_exps.=CPU". Every expert you move back onto the GPU is faster but costs VRAM, so tune the regex to fill your card exactly. --cache-type-k q4_0 quantizes the key cache to shrink the memory that long context consumes.

Multi-GPU with vLLM

If you have a multi-GPU server, vLLM gives you far higher throughput and a production-grade OpenAI-compatible server. It shards the model across GPUs with tensor parallelism:

vllm serve moonshotai/Kimi-K3 \
  --tensor-parallel-size 8 \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

Set --tensor-parallel-size to your GPU count, --max-model-len to the context you actually need (higher costs KV-cache memory), and lower --gpu-memory-utilization from its default if you hit out-of-memory on load. For a model this large you need enough aggregate VRAM to hold the weights at your chosen precision; a single node of eight 80GB cards is the usual floor for a higher-bit quant.

On a Mac: MLX and LM Studio

Apple Silicon is a quietly excellent option here because the GPU and CPU share one pool of unified memory, so a 256 or 512GB Mac Studio can hold a model that no single discrete GPU could. The native path is MLX:

pip install mlx-lm
mlx_lm.generate --model mlx-community/Kimi-K3-4bit --prompt "Explain MoE routing."
# OpenAI-compatible server:
mlx_lm.server --model mlx-community/Kimi-K3-4bit --port 8080

If you would rather not touch a terminal, LM Studio wraps both llama.cpp (for GGUF) and MLX in a desktop app, lets you search and download a quant, and exposes a local OpenAI-compatible server at http://localhost:1234/v1 with one toggle. Same quant tiers apply; pick the largest that fits your unified memory.

Serve an OpenAI-compatible API

Every engine above can present the OpenAI API, which is the point: your existing agents and SDKs drive a local model by changing one URL. The llama.cpp server exposes /v1/chat/completions on the host and port you set. Call it like any OpenAI endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Say hi in one line."}]}'

Point any OpenAI client at it by setting the base URL to http://127.0.0.1:8080/v1 and any string as the API key. Coding agents and frameworks that accept an OpenAI-compatible base URL will then run entirely against your local Kimi K3.

Tool calling

Kimi K3 supports tool calling, and the mechanism is the chat template. In llama.cpp, pass --jinja so the server uses the model's embedded template, which knows how to format tool definitions and parse tool calls; without it, tool output is malformed. Then send an OpenAI-style tools array in the request and read the tool call back from the response. In vLLM, enable it explicitly with --enable-auto-tool-choice and the matching --tool-call-parser for the model family. Because the endpoint is OpenAI-compatible, an agent framework passes its tool schemas unchanged.

Context and KV cache

Long context is not free: the KV cache grows with the number of tokens you keep, and on a big model it can rival the weights for memory. Two levers help. Set -c (or --max-model-len in vLLM) only as high as you actually need rather than the model's maximum, and quantize the cache with --cache-type-k q4_0 and, with flash attention enabled, --cache-type-v q4_0 in llama.cpp, or --kv-cache-dtype fp8 in vLLM. Enabling flash attention (-fa on) also reduces cache memory and speeds attention. Only push toward the model's full context window if you have measured that the KV cache fits.

Sampler settings

Use the model author's recommended sampling rather than defaults. For Kimi K3 the documented starting point is a temperature around 1.0 with top-p 0.95 for general use, moving toward top-p 1.0 for agentic and tool-heavy runs, and a reasoning_effort control for how much the model thinks before answering. Confirm the current numbers on the model card, since reasoning models are sensitive to sampler settings and wrong values cause loops or shallow answers.

Speed to expect

Set expectations by where the weights live. On a single 24GB GPU paired with a couple hundred gigabytes of system RAM and the experts offloaded, a dynamic low-bit quant runs at a few tokens per second, bounded by RAM bandwidth rather than the GPU. A high-memory Mac lands in a similar-to-better range depending on the chip. A multi-GPU server that holds the whole model in VRAM is an order of magnitude faster and is what you want for anything interactive or multi-user. If you are getting well under a token per second, you are spilling to SSD and should drop to a smaller quant or add RAM.

Troubleshooting

• Out of memory on load. Drop to a smaller quant, lower context (-c or --max-model-len), offload more experts to CPU with -ot, quantize the KV cache, or in vLLM lower --gpu-memory-utilization.
• Gibberish output. Almost always a wrong or missing chat template. In llama.cpp pass --jinja; make sure you pulled the instruct model, not a base checkpoint; check you are not doubling special tokens.
• Painfully slow. You are spilling to disk. Use a smaller quant, add RAM, enable flash attention, or move more layers onto the GPU if VRAM allows.
• Multi-part GGUF will not load. Point at the first shard, or merge with llama-gguf-split --merge. Ollama in particular usually wants a single merged file.
• Context seems truncated. Your engine defaulted low. Ollama defaults to a 2048 context; raise it. In llama.cpp set -c to what you need.

FAQ

What hardware do I need? Memory sized to the total parameters, not the active ones. That means a multi-GPU server, a big-RAM workstation running a dynamic low-bit quant with experts offloaded to system RAM, or a high-memory Apple Silicon machine. A single consumer GPU alone will not hold it.

Can I use Ollama? Ollama is best up to about 32B and a single GGUF file. For Kimi K3's size use llama.cpp, which supports expert CPU offload and multi-part files, or vLLM across GPUs.

Which quant? If memory constrained, a dynamic low-bit quant is what makes it fit while staying close to full precision. If you have the memory, Q8 is near lossless. Match the quant file size to your combined VRAM plus RAM.