Run Gemma 4 Locally: Hardware, Runtimes, and Which Size to Pick

by goyostudio

Gemma 4 is Google DeepMind's open-weights model family, released April 2, 2026 under Apache 2.0. The license change matters: Gemma 1 through 3 shipped under Google's custom Gemma Terms of Use with a use-restriction policy, while Apache 2.0 is a plain permissive OSI license that allows commercial use, redistribution, and derivative models with only attribution and license-notice obligations. Weights download free from Hugging Face, Kaggle, and Ollama, and there is no per-token cost because you run them yourself. Five sizes cover everything from a phone to a multi-GPU server, all natively multimodal (text plus image, with audio on the smaller three) and multilingual across 140+ languages.

The same weights run everywhere, so the only real decision is which size your hardware can hold. That is what this guide covers: how the five sizes map to memory, what quantization actually buys you, and how to serve the result.

## The lineup

  • E2B — 2.3B effective parameters, 128K context. The smallest member; under 5 GB at 4-bit, so it runs on a laptop CPU or a modest GPU with no accelerator required.
  • E4B — 4.5B effective parameters, 128K context. The practical default, and what the bare gemma4 tag resolves to. Roughly 5-8 GB at 4-bit.
  • 12B — unified multimodal, 256K context, around 7 GB at 4-bit. The largest size that still accepts audio input.
  • 26B A4B — a Mixture-of-Experts with 25.2B total parameters that routes each token to 8 of 128 experts plus one shared expert, activating 3.8B per token. Near-31B quality at roughly 4B speed, 256K context.
  • 31B — dense, 256K context, the most capable of the family. About 17 GB at 4-bit and 58 GB in bf16.

The E models use Per-Layer Embeddings to keep active compute far below the loaded parameter count, which is why they fit tiny memory budgets. They are also, along with the 12B, the only sizes that accept audio; the 26B and 31B are text plus image. Every size reads images, and video is processed as sampled frames.

// note: 26B A4B is a Mixture-of-Experts, not a dense 26B. It is faster than its size suggests, but it still loads all 25.2B parameters into memory — plan RAM and VRAM around the full weight, not the 3.8B active per token.

## VRAM math

Weights scale at roughly 2 bytes per parameter in bf16 and roughly 0.6 bytes per parameter at 4-bit. Approximate weights-only footprints, given as bf16 / 8-bit / 4-bit:

  • E2B — ~10 / 5 / 3 GB
  • E4B — ~15 / 7.5 / 5 GB
  • 12B — ~24 / 12 / 7 GB
  • 26B A4B — ~48 / 25 / 15 GB
  • 31B — ~58 / 30 / 17 GB

Treat those as floors, not budgets. They cover weights only. The KV cache sits on top and grows with context length and with the number of requests you batch, and at a long 256K context it can add several GB — for a heavily batched server it can consume more memory than the model itself. Leave headroom, and cap the context explicitly rather than hoping the default is small enough.

## Quantization in plain words

Quantization stores each weight in fewer bits. Going from bf16 to 4-bit cuts the weight footprint by roughly 3x in practice, which is the difference between a 31B that needs a data-center card and a 31B that fits on a 24 GB consumer GPU. The cost is precision: rounding a bf16 checkpoint down after training is a lossy operation, and naive post-training quantization is where most of the perceived quality drop comes from.

Google sidesteps that with Quantization-Aware Trained checkpoints, published as google/gemma-4-<size>-it-qat-q4_0-gguf. These were fine-tuned to tolerate 4-bit weights, so they recover most of the bf16 quality that a naive int4 conversion loses while still costing about a third of the memory. If you are running 4-bit at all, start with a QAT build rather than quantizing a bf16 repo yourself.

bf16 still matters in two cases. First, if you want 8-bit or full precision, you need the safetensors repo (google/gemma-4-E4B-it) and your own quantization step — QAT GGUFs are int4 only and cannot be up-cast. Second, fine-tuning: LoRA on the 26B MoE is done in bf16, and QLoRA loads the base in 4-bit but still trains adapters in higher precision.

// note: QAT weights are already int4. Do not re-quantize them — stacking a bitsandbytes 4-bit config on top corrupts the values and tanks quality. Load them as-is.

## Ollama: the shortest path

Ollama is the fastest way to get a first token. One command pulls the weights and drops you into an interactive chat; /bye exits, /show info prints the effective modelfile. The E2B download is about 7 GB on disk.

# Smallest size, CPU-friendly
ollama run gemma4:e2b

// note: The bare ollama run gemma4 pulls the E4B build (the tag marked latest, ~9.6 GB), not E2B. Ask for gemma4:e2b explicitly when you are tight on memory.

ollama pull fetches a tag without starting a chat. The available tags are gemma4:e2b, gemma4:e4b, gemma4:12b, gemma4:26b (the A4B MoE, ~18 GB), and gemma4:31b. For a straightforward dense mid-size, gemma4:12b is ~7.6 GB.

# Download without starting a chat
ollama pull gemma4:31b
ollama pull gemma4:e4b

On Apple Silicon, the -mlx tags (gemma4:e2b-mlx through gemma4:31b-mlx) use Apple's MLX runtime for faster, more memory-efficient inference than the default GGUF path, at comparable download sizes. They only run on M-series Macs — on x86 or NVIDIA the runtime cannot load them, so use the plain tags there.

# Apple Silicon build
ollama run gemma4:e4b-mlx

The Ollama daemon exposes a REST API on port 11434 whenever it runs. /api/generate takes a single prompt and /api/chat takes a messages array; there is also an OpenAI-compatible surface at http://localhost:11434/v1/chat/completions for drop-in clients. By default /api/generate streams newline-delimited JSON, so pass "stream": false when you want one complete response.

# Local Ollama API
curl http://localhost:11434/api/generate -d '{"model":"gemma4","prompt":"roses are red","stream":false}'

## llama.cpp and GGUF: portable and CPU-capable

llama.cpp runs Gemma 4 on CPU, GPU, or a hybrid of both, which makes it the right tool when you have RAM but not VRAM. The -hf flag downloads a GGUF from Hugging Face and runs it in one step, with no manual conversion; downloads cache under ~/.cache/llama.cpp. It needs a repo that actually contains .gguf files, so point it at ggml-org/gemma-4-E2B-it-GGUF or a QAT repo — not the raw safetensors repo.

# Download and run a GGUF in one command
llama-cli -hf ggml-org/gemma-4-E2B-it-GGUF -p "Write a haiku about the sea."

Two flags control memory. -ngl/--n-gpu-layers offloads that many transformer layers to the GPU; -ngl 99 pushes all of them and silently caps at the real layer count. Without it, everything runs on CPU, which is slow for the 12B and larger. -c/--ctx-size sets the context window in tokens, and the default is small — you have to ask for a long context, and pay for it in KV cache.

# 32K context, all layers on GPU
llama-cli -hf ggml-org/gemma-4-12B-it-GGUF -c 32768 -ngl 99 -p "Summarize this file."

llama-server loads the GGUF once and keeps it resident: a browser chat UI at http://localhost:8080 plus an OpenAI-compatible API at http://localhost:8080/v1/chat/completions. Any OpenAI SDK works against that base URL with a placeholder key.

# Persistent local server on port 8080
llama-server -hf ggml-org/gemma-4-E4B-it-GGUF -c 8192 -ngl 99

The QAT int4 builds work with the same -hf mechanism, and this is where they pay off most — best quality per byte on a memory-constrained machine.

# Google's quantization-aware trained int4 weights
llama-cli -hf google/gemma-4-E4B-it-qat-q4_0-gguf -p "Explain QAT in one sentence."

## vLLM: throughput and multi-user serving

vLLM is the choice when more than one person or process hits the model. vllm serve loads a Hugging Face repo and exposes an OpenAI-compatible server on port 8000, and its paged attention is what makes the 26B and 31B practical under concurrent load. --max-model-len caps the context and, with it, the KV cache.

# OpenAI-compatible server on port 8000
vllm serve google/gemma-4-31B-it --max-model-len 16384

// note: vLLM sizes the KV cache from --max-model-len at startup. Omit it and vLLM may try to reserve the full 256K context and fail with an out-of-memory error before serving a single request. llama.cpp has the same failure mode: -c 262144 on a small GPU allocates a cache that dwarfs the weights and aborts before the first token. Start around 8192 and raise it.

For a bf16 31B (~58 GB), --tensor-parallel-size shards the weights so it fits on two 40 GB cards. The value must divide the attention-head count evenly and the GPUs should be identical, or it fails at load time. --gpu-memory-utilization (0-1) caps how much of each card vLLM claims, leaving room for other processes.

# 31B in bf16 across two GPUs
vllm serve google/gemma-4-31B-it --tensor-parallel-size 2 --gpu-memory-utilization 0.90 --max-model-len 16384

If your app uses tools or Gemma 4's thinking mode, add the parsers. --enable-auto-tool-choice lets the model emit tool calls, while --tool-call-parser gemma4 and --reasoning-parser gemma4 split those calls and the thinking segment out of the raw text into structured tool_calls and reasoning_content fields. Skip the reasoning parser and the raw thought block stays inside the assistant message, which means you ship the model's private reasoning to your users.

# Tool calling plus reasoning parsing
vllm serve google/gemma-4-31B-it --enable-auto-tool-choice --tool-call-parser gemma4 --reasoning-parser gemma4 --chat-template examples/tool_chat_template_gemma4.jinja

All three runtimes speak the same /v1/chat/completions schema, so one client works against any of them — only the port changes (vLLM 8000, llama-server 8080, Ollama 11434). The model string must match exactly what the server loaded; for vLLM that is the full repo id, not a short alias.

# Same request shape for vLLM, llama-server, and Ollama
curl http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' -d '{"model":"google/gemma-4-31B-it","messages":[{"role":"user","content":"Explain MoE in one sentence."}]}'

## Disk, not just memory

A single 31B checkpoint is 20+ GB, and with five sizes plus quantized variants a model directory fills a disk quietly. Ollama stores blobs under ~/.ollama/models, Transformers and huggingface-cli cache to ~/.cache/huggingface/hub, and llama.cpp's -hf downloads land in ~/.cache/llama.cpp. Set OLLAMA_MODELS or HF_HOME to relocate them off a small root disk, and reclaim space with ollama rm <tag>.

# Audit what the caches are holding
du -sh ~/.ollama/models
du -sh ~/.cache/huggingface/hub

## Two settings to fix on day one

Google recommends temperature 1.0, top_p 0.95, and top_k 64 for Gemma 4. That temperature is higher than the 0.7 many models default to, so a low-temperature preset copied from another model makes Gemma 4 feel flat and repetitive. In the Ollama REPL, change them live.

# Recommended sampling settings
/set parameter temperature 1.0
/set parameter top_p 0.95
/set parameter top_k 64

Second, if you turn on thinking mode with the <|think|> token, raise your token limit. A reasoning pass can run 4,000+ tokens before the final answer, and a limit sized for the answer alone means the model spends its whole budget reasoning and returns something truncated or empty. Steer the depth from the system prompt when you need speed.

# Thinking mode with a bounded budget
llama-cli -hf ggml-org/gemma-4-E4B-it-GGUF -sys "<|think|>Think briefly, focus on key steps." -n 2048 -p "Is 2027 prime?"

## Which size for your machine

  • 8 GB laptop, no discrete GPU → gemma4:e2b at 4-bit (~3 GB), running on CPU. On an M-series Mac use gemma4:e2b-mlx.
  • 16 GB laptop or an 8 GB GPU → gemma4:e4b (~5 GB at 4-bit) as the everyday default, or gemma4:12b (~7 GB at 4-bit) when you want the 256K context and better quality.
  • 12-16 GB GPU → 12B at 4-bit with real room for the KV cache. A 16 GB card is also the QLoRA fine-tuning floor for E4B.
  • 24 GB GPU → 31B via Google's QAT int4 checkpoint (~17 GB), leaving roughly 7 GB for the KV cache. Cap the context around 16K.
  • 48-80 GB GPU → 31B in bf16 (~58 GB) at full precision, or 26B A4B (~48 GB bf16) when throughput matters more than peak quality.
  • Two 40 GB cards → 31B in bf16 with --tensor-parallel-size 2.
  • Need audio input → E2B, E4B, or 12B only; the 26B and 31B are image-only.
  • Runtime: Ollama to try it, llama.cpp for CPU or single-user local use, vLLM for anything multi-user.

// related cheat sheets