How to Run DeepSeek Locally in 2026: R1 Distills, the 671B MoE, GGUF, Ollama, llama.cpp & vLLM

How do you run DeepSeek locally?

Pick the variant that fits your hardware. The realistic local option for most people is the DeepSeek-R1 distill family — dense models from 1.5B to 70B that run like the Qwen and Llama models they're distilled from: the 8B fits an 8GB card, and the 32B fits a single RTX 4090 (~20GB at a 4-bit quant). The full 671B-parameter DeepSeek-V3/R1 is a Mixture-of-Experts model with only 37B active per token — powerful, but its 4-bit GGUF is ~404GB, so it needs a large unified-memory machine or a GPU cluster (or Unsloth's dynamic 1.58-bit build at ~131GB). Everything is MIT-licensed, so nothing leaves your machine and there's no per-token bill. Below: exactly which model, what hardware, which tool, and the commands.

Which DeepSeek model should you run locally?

DeepSeek's lineup causes more confusion than any other open model, so start here. Two families matter:

For almost everyone, the answer is a distill: R1-Distill-Qwen-32B for max quality on one 24GB card, or R1-Distill-Llama-8B / Qwen-7B for anything smaller. The full 671B (V3, the R1 reasoning model, the V3.1 hybrid, or the V3.2-Exp sparse-attention build) is worth knowing about but is a data-center model, not a desktop one.

The hardware reality: VRAM by model and quant

The distills are what you'll actually run. Practical targets at a 4-bit (Q4_K_M) GGUF — directional estimates, not official figures:

Two rules of thumb: leave 20–30% of VRAM free for the KV cache, and on Apple silicon count on unified memory (a 128GB Mac runs the 70B distill).

GGUF and quantization: what Q4_K_M actually buys you

For local use you download a GGUF — a single-file, quantized build that llama.cpp and Ollama load directly — not the original weights.

• Q4_K_M is the default: it cuts VRAM roughly 60% versus FP16 with negligible quality loss. Start here.
• Q5_K_M / Q6_K trade a few GB for a slight quality bump.
• Q8_0 is near-lossless but large.
• Q3 and below save memory but degrade — a fallback, not a default.

The practical move is the same as for any model: pick the largest model that fits at Q4, not a smaller model at Q8. A 32B distill at Q4 beats an 8B at Q8 on almost everything.

Can you run the full 671B DeepSeek?

Yes — but you need to be clever about it. The native FP8 weights are ~671–700GB, and even a Q4_K_M GGUF is ~404GB, which realistically means a ~512GB unified-memory box or a multi-GPU cluster. The trick that makes it runnable off a cluster is Unsloth's dynamic quantization, which shrinks the 671B from ~720GB to ~131GB (about 80% smaller) while keeping it functional:

The insight is that calibration matters more than bit-count: Unsloth keeps the first few layers and attention modules at 4–6 bit and pushes only the bulk MoE layers to ~1.5-bit. A naive all-layer 1-bit quant of DeepSeek produces loops and gibberish — the selective version doesn't. The 131GB build runs in ~160GB of VRAM (2× H100) at ~140 tokens/sec, or on a big-RAM box with the MoE experts offloaded to CPU. (Quant sizes shift between R1 revisions — confirm current numbers on the Unsloth Hugging Face repo.)

Which hardware actually runs which DeepSeek?

Which tool: Ollama, llama.cpp, vLLM, or SGLang?

For a quantized distill on your own box, Ollama is the fastest start. For serving the native FP8 671B on a cluster, SGLang (DeepSeek's own recommendation) or vLLM.

How to run it, step by step

Option A — Ollama (easiest):

ollama run deepseek-r1:8b     # an R1 distill
ollama run deepseek-r1:32b    # the 32B distill — fits a 4090

You get a chat prompt and an OpenAI-compatible endpoint at http://localhost:11434/v1.

Option B — llama.cpp (control): download a distill GGUF (e.g. a Q4_K_M), then:

llama-server -m DeepSeek-R1-Distill-Qwen-32B-Q4_K_M.gguf -c 8192 -ngl 99

-ngl 99 offloads all layers to GPU; -c 8192 sets context — lower it if you hit out-of-memory. For the full 671B on a big-RAM box, keep the MoE experts on CPU (llama.cpp's MoE-offload flags — confirm the current syntax against the Unsloth tutorial).

Option C — vLLM / SGLang (native, cluster):

vllm serve deepseek-ai/DeepSeek-R1 --tensor-parallel-size 8 --trust-remote-code

V3/R1 ship already in FP8, so don't re-quantize them; serve across 8 GPUs with tensor parallelism.

Performance: what to expect

On a single RTX 4090, rough single-user throughput by distill size:

The honest note: a full 671B on a big-RAM box (experts offloaded to CPU) is usable for batch and agentic work, not instant chat — the memory bandwidth caps it. The distills are where you get snappy interactive speed.

What can go wrong when self-hosting DeepSeek

• KV-cache out-of-memory at long context. 128K context is a huge KV cache — keep context modest, especially on the 671B.
• Undersized memory → disk offload. If RAM+VRAM can't cover the quant, it runs off disk and crawls. Size the box to the model.
• Confusing total vs. active params. The 671B uses 37B per token for speed, but you still need memory for the whole 671B. It's fast, not small.
• Downloading BF16 by habit. Grab the GGUF (or serve the native FP8), not the raw BF16 weights.
• Re-quantizing FP8 weights. V3/R1 are already FP8 — don't pass a quantization flag over them.
• Unofficial mirrors. Pull weights from DeepSeek's or Unsloth's official repos, not random re-uploads.

When to self-host vs. use the API

Self-host when data can't leave your environment (regulated industries, private code), when you want zero per-token cost at volume, or when you need a distill running on hardware you already own. Use the hosted API when you want full-671B quality without ~130–400GB of memory, when volume is low enough that per-token pricing beats a cluster, or when you're just building and don't want to operate infrastructure.

Related reading

• How to run Kimi K3 locally — the same playbook for a trillion-parameter MoE model.
• How to run Qwen 3 locally — the dense/MoE family the DeepSeek distills are built on.
• The 2026 AI infrastructure stack — the production serving stack around models like these.

Build the infra skills to run models like this

Self-hosting a model like DeepSeek — especially the 671B — is real infrastructure engineering: quantization, multi-GPU serving, KV-cache and memory management, reliability. Dexity's AI Platform Engineering course builds exactly that muscle, hands-on, so standing up a local DeepSeek deployment is a project you run, not just read about.

Model specs, VRAM figures, and tooling verified against DeepSeek's V3 report, the R1 and V3.2-Exp model cards, Unsloth's dynamic-GGUF guide, the Ollama library, and SGLang's DeepSeek docs (verified September 2026). Distill VRAM and tokens/sec are directional estimates; open-model versions, quant sizes, and tooling move quickly — confirm current builds before committing hardware. · Dexity.com