How to Run Qwen 3 Locally in 2026: GGUF, llama.cpp, vLLM & the Hardware You Actually Need

How do you run Qwen 3 locally?

Pick the variant that fits your GPU, download a GGUF quant, and serve it with Ollama, llama.cpp, or vLLM. That's the whole job — the only real decision is model size. Qwen 3's 8B model runs in ~5–6 GB of VRAM (an RTX 3060 or a 16 GB Apple silicon Mac), the 32B dense model needs ~20 GB (RTX 4090 territory), and the 30B-A3B Mixture-of-Experts model is the local sweet spot — it fits a single 24 GB card and delivers ~25 tokens/sec because only ~3B of its 30B parameters activate per token. The whole Qwen 3 family is Apache 2.0, so nothing leaves your machine and there's no per-token bill. Below: the exact hardware, quant, and tooling choices, with commands.

Which Qwen 3 model should you run locally?

The family is unusually broad, which is why it's become the default open model — there's a size for every machine. What matters for local use is dense vs. MoE and total vs. active parameters.

For most people the answer is Qwen3-8B (fast, fits anything) or Qwen3-30B-A3B (near-32B quality at a fraction of the compute per token). The dense 32B is the highest-quality single-card option but leaves almost no headroom for context. The 235B-A22B is a data-center model — worth knowing about, not something you run on one desktop.

The hardware reality: VRAM by model and quant

The single number that decides everything is VRAM. These are practical targets at a 4-bit (Q4_K_M) GGUF quant, the most common local choice:

Two rules of thumb: leave 20–30% of your VRAM free for the KV cache (the model's working memory for context), and on Apple silicon, count on unified memory rather than a discrete GPU — a 32 GB+ Mac runs the 30B-A3B comfortably.

GGUF and quantization: what Q4_K_M actually buys you

You almost never download the original BF16 weights for local use — they're twice the size for quality you can't perceive at chat-length prompts. Instead you download a GGUF file: a single-file, quantized format that llama.cpp and Ollama load directly.

• Q4_K_M is the default: it cuts VRAM roughly 60% versus FP16 with negligible quality loss for most tasks. Start here.
• Q5_K_M / Q6_K trade a few more GB for a slight quality bump — worth it if you have headroom.
• Q8_0 is near-lossless but large; use it only if VRAM is plentiful.
• Q3 and below save memory but degrade noticeably — a fallback, not a default.

The practical move: pick the largest model that fits at Q4_K_M rather than a smaller model at Q8. A 30B-A3B at Q4 beats an 8B at Q8 on almost everything.

Which tool: Ollama, llama.cpp, vLLM, or SGLang?

For 95% of local use, Ollama is the right starting point — it installs fast, pulls models with one command, and exposes an OpenAI-compatible API so your existing code works unchanged. Move to vLLM or SGLang only when you're past one user and need real throughput.

How to run it, step by step

Option A — Ollama (easiest):

# install Ollama, then:
ollama run qwen3:8b
# or the MoE sweet spot:
ollama run qwen3:30b-a3b

That's it — you get a chat prompt and an OpenAI-compatible endpoint at http://localhost:11434/v1.

Option B — llama.cpp (control): download a Qwen3 GGUF (e.g. a Q4_K_M build), then:

llama-server -m qwen3-30b-a3b-Q4_K_M.gguf -c 8192 -ngl 99

-ngl 99 offloads all layers to GPU; -c 8192 sets context length — lower it if you hit out-of-memory.

Option C — vLLM (serving a team):

vllm serve Qwen/Qwen3-8B --max-model-len 32768

vLLM loads the full (unquantized) weights by default, so size your GPU accordingly, and gives you high-throughput batched inference behind an OpenAI-compatible API.

Performance: what to expect

Speed depends on model, quant, and hardware, but rough single-user expectations on a modern consumer GPU (RTX 4090):

The MoE model's trick is visible here: 30B-A3B runs far faster than the dense 32B despite similar quality, because only ~3B parameters fire per token. On Apple silicon, expect lower throughput than a 4090 but enough for interactive use on the 8B–30B range.

What can go wrong when self-hosting Qwen 3

• KV-cache out-of-memory. The 32B's ~20 GB GGUF leaves only 3–4 GB on a 24 GB card, so a long prompt overflows and crashes. Fix: use the 30B-A3B MoE instead, drop the quant, or shorten context.
• Confusing total vs. active params. A 30B MoE uses ~3B per token for speed, but you still need VRAM for the whole 30B. It's fast, not small.
• Downloading BF16 by habit. Grab the GGUF quant, not the originals, unless you're serving with vLLM.
• Context creep. Native context is 32K on the big MoE; pushing to 131K needs YaRN scaling and much more memory. Don't set a huge -c you can't feed.

When to self-host vs. use the API

Self-host when data can't leave your environment (regulated industries, private code, PII), when you want zero per-token cost at high volume, or when you need full control over the model and its uptime. Use the hosted API when you want the 235B-class or Qwen3-Max quality without owning GPUs, when volume is low enough that per-token pricing is cheaper than hardware, or when you don't want to operate infrastructure. Most teams end up doing both: a local 8B or 30B-A3B for the bulk of requests, and an API for the occasional frontier-grade query.

Related reading

• How to run Kimi K3 locally — the same playbook for a trillion-parameter MoE model.
• The 2026 AI infrastructure stack — what a production serving stack looks like around models like these.
• AI infrastructure & platform engineer, by the numbers — who's hiring for exactly this skill, and what it pays.

Build the infra skills to run models like this

Self-hosting an open model well is real infrastructure engineering — quantization, GPU memory and KV-cache management, multi-GPU serving, and reliability. Dexity's AI Platform Engineering course builds exactly that muscle, hands-on, so standing up a local Qwen 3 deployment is a project you run, not just read about.

Model specs, VRAM figures, and tooling verified against the official Qwen 3 blog and Spheron's Qwen3 local-GPU guide (verified September 2026). Open-model versions, quant sizes, and tooling move quickly — confirm current builds before committing hardware. · Dexity.com