Local model guide · updated July 2026 · 11 min read

How to Run Gemma 3 Locally

Gemma 3 is Google's dense open family, and it is the most forgiving one to run because the sizes are ordinary dense models with no exotic memory strategy required. It spans a phone-sized 1B up to a 27B that sits on a single 24GB GPU, and the 4B and larger models can see images. This guide picks a size for your hardware, runs it on every engine, and covers vision, the recommended sampler, and the double-BOS gotcha that trips people up.

Pick a size

The 27B at 4-bit is the sweet spot for a single high-end consumer card, and the 4B is the smallest model that still sees images, which makes it a strong choice for a laptop.

Run with Ollama

curl -fsSL https://ollama.com/install.sh | sh
ollama run gemma3:27b
# smaller: gemma3:12b, gemma3:4b, gemma3:1b

As always with Ollama, raise the context above the low default for long inputs with a Modelfile setting num_ctx.

Run with llama.cpp

Build llama.cpp (steps here), then serve the text model:

./llama.cpp/llama-server \
  -hf unsloth/gemma-3-27b-it-GGUF:Q4_K_M \
  --jinja \
  -ngl 99 \
  -c 32768 \
  --host 127.0.0.1 --port 8080

Images and vision

The 4B and larger models are multimodal. Vision needs a second file, the multimodal projector, which ships inside the GGUF repo as an mmproj file, plus llama.cpp's multimodal binary:

./llama.cpp/llama-mtmd-cli \
  -hf unsloth/gemma-3-27b-it-GGUF:Q4_K_M \
  --mmproj mmproj-F16.gguf \
  -ngl 99 \
  --image photo.jpg \
  -p "Describe what is in this image."

Ollama also accepts image input on the vision-capable Gemma 3 tags, so you can pass an image path in the chat and it handles the projector for you.

Serve with vLLM

vllm serve google/gemma-3-27b-it \
  --max-model-len 32768 \
  --gpu-memory-utilization 0.90

On a Mac

pip install mlx-lm
mlx_lm.server --model mlx-community/gemma-3-27b-it-4bit --port 8080

LM Studio also lists every Gemma 3 size and gives you a local OpenAI-compatible server without the terminal.

Sampler and the BOS gotcha

Gemma 3 wants a distinctive sampler: a temperature of 1.0 with top-k 64, top-p 0.95, min-p 0.0, and a repetition penalty of 1.0. The most common quality problem is a double beginning-of-sequence token. Gemma's template adds a <bos>, and if your client also adds one you get two, which degrades output. When you run through llama.cpp with --jinja or through Ollama this is handled; if you build prompts by hand, add the BOS only once.

Tool calling

Gemma 3 supports tool calling through its chat template. Pass --jinja in llama.cpp so the template formats tools correctly, and drive it through the OpenAI-compatible endpoint with a standard tools array. In vLLM, enable auto tool choice and use the parser matching the Gemma template as noted on the model card.

Troubleshooting

• Degraded or odd output. Usually a double BOS. Use --jinja or Ollama, or add the beginning-of-sequence token only once.
• Truncated long prompts. Ollama's default context is 2048. Set num_ctx, or -c in llama.cpp.
• Vision does nothing. You are missing the projector. Use llama-mtmd-cli with --mmproj and a 4B or larger model.
• Out of memory. Drop a quant tier or a model size; the 27B at Q8 needs more than 24GB.

FAQ

What GPU runs the 27B? A 24GB card at 4-bit, with weights around 18 to 20 GB. The 12B fits 12 to 16 GB, the 4B fits 8 GB, and the 1B runs on very small devices.

Can it see images? Yes, the 4B and larger models. Use the mmproj projector file with llama-mtmd-cli, or a vision tag in Ollama.

Sampler? Temperature 1.0, top-k 64, top-p 0.95, min-p 0.0, repeat penalty 1.0, and avoid a double BOS.