Local model guide · updated July 2026 · 10 min read

How to Run GLM Locally

GLM from z.ai has become a favorite for local agentic coding: MIT-licensed mixture-of-experts models that punch above their active size and drive tools well. The compact GLM-4.5 Air runs on a single high-end card with offload, the flagship GLM-4.6 wants a big-RAM box, and the GLM-5 series is server-scale. This guide sizes each and runs them across llama.cpp and vLLM, with a focus on wiring GLM into a coding agent.

Which GLM

Air is the one to start with. It keeps a small active-parameter count, so it runs fast, and a dynamic 4-bit quant fits a single high-end card once you offload the experts to RAM. Step up to 4.6 or the 5 series when you have the memory and want the top of the quality curve.

GLM-4.5 Air on a GPU

Build llama.cpp (steps here), then serve Air with expert offload:

./llama.cpp/llama-server \
  -hf unsloth/GLM-4.5-Air-GGUF:Q4_K_M \
  --jinja \
  -ngl 99 \
  -ot ".ffn_.*_exps.=CPU" \
  -c 32768 \
  --host 127.0.0.1 --port 8080

On a 64GB or larger Mac you can drop the offload flag and keep the whole model in unified memory.

GLM-4.6 and 5 with offload

The larger variants follow the big-model pattern: a dynamic low-bit quant plus -ot ".ffn_.*_exps.=CPU" to keep experts in system RAM, matched to your combined VRAM plus RAM. Merge any multi-part GGUF with llama-gguf-split --merge. Expect the same memory-bandwidth-bound throughput as other 600B-plus-class models on a single-GPU-plus-RAM box.

./llama.cpp/llama-server \
  -hf unsloth/GLM-4.6-GGUF:UD-Q2_K_XL \
  --jinja -ngl 99 -ot ".ffn_.*_exps.=CPU" \
  --cache-type-k q4_0 -c 16384 --host 127.0.0.1 --port 8080

Serve with vLLM

vllm serve zai-org/GLM-4.6 \
  --tensor-parallel-size 8 \
  --max-model-len 131072 \
  --enable-auto-tool-choice \
  --tool-call-parser glm

Use the GLM tool-call parser named on the model card for your checkpoint, and set --tensor-parallel-size to your GPU count.

Wire it into a coding agent

GLM's strength locally is agentic coding, so the payoff is connecting it to an agent. Start the server, then point your coding agent at the local endpoint:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"messages":[{"role":"user","content":"Refactor this function for readability."}]}'

Any agent or framework that accepts an OpenAI base URL will then run against your local GLM: set the base URL to http://127.0.0.1:8080/v1 and any string as the key. Make sure tool calling is enabled so the agent's file and shell tools format correctly.

Tool calling

Enable tools with the GLM parser in vLLM and --enable-auto-tool-choice, or --jinja in llama.cpp for the model's template. Because the endpoint is OpenAI-compatible, the agent passes its tool schemas unchanged, which is what makes GLM a clean drop-in for a local coding loop.

Troubleshooting

• Out of memory. Offload more experts with -ot, drop to a smaller dynamic quant, quantize the KV cache, and lower context.
• Slow on the larger variants. Expected on single-GPU offload; a few tokens per second. Add GPUs or RAM for speed.
• Tool calls not parsed. Use the GLM parser in vLLM, or --jinja in llama.cpp.
• Multi-part GGUF will not load. Merge with llama-gguf-split --merge or point at the first shard.

FAQ

Which GLM runs on a consumer GPU? GLM-4.5 Air, at 4-bit with expert offload on a 24GB card plus RAM, or on a high-memory Mac. The full 4.6 and the 5 series need a big-RAM box or a server.

Good for coding agents? Yes, GLM is a strong agentic coder. Serve it with an OpenAI endpoint and point your agent at it.

License? The open-weight GLM models are MIT; confirm the specific checkpoint's card.