Deploying DeepSeek-V3 (671B) and DeepSeek-Coder-V2 (236B) locally exposes the definitive architectural trap of modern LLM hosting:
Total parameter volume dictates where the model physically lives, but active parameter count dictates how fast it computes.
While DeepSeek-V3 only computes 37B active parameters per token (and Coder-V2 computes just 21B), your GPU memory controllers cannot ignore the inactive experts. All 236B to 671B parameter weights must reside permanently in addressable memory.
Here is what physical testing across single-GPU, dual-GPU, and Apple Silicon architectures actually reveals.
1. The MoE Memory Trap: Active vs. Total Parameters
When hardware sizing traditional dense foundation models—like Llama 3.1 or Qwen 2.5—every weight is engaged during every forward pass.
In contrast, DeepSeek models utilize sparse Mixture-of-Experts routing:
- DeepSeek-Coder-V2: 236B total parameters, but only 21B active per token.
- DeepSeek-V3: 671B total parameters, but only 37B active per token (routed across 8 experts + 1 shared expert).
Because routing decisions change dynamically at token $N+1$, you cannot predict which expert will be needed next. Therefore, every single expert matrix must remain pre-loaded inside VRAM or system RAM.
2. Multi-Head Latent Attention (MLA): The KV Cache Savior
Normally, context expansion is the primary killer of local GPUs. In standard Grouped-Query Attention (GQA), running a 128K context window can easily consume 30GB+ of VRAM just for context buffers.
DeepSeek resolves this context bloat through Multi-Head Latent Attention (MLA):
- Instead of caching separate, uncompressed Keys and Values, MLA low-rank compresses them into a single latent vector.
- The practical payoff: For DeepSeek-Coder-V2 (236B), an active 32,768-token context window consumes just ~1.4 GB of VRAM, compared to over 18GB on standard dense models.
3. Empirical Hardware Benchmarks: Real Tokens/Sec
| Configuration | Target Model | Quant / Size | Token Velocity | Practical Verdict |
|---|---|---|---|---|
| Single RTX 3060 / 4060 Ti | Coder-V2-Lite | Q4_K_M (9.4GB) | 52 - 64 tok/s | Pure VRAM execution, ideal for VS Code |
| Dual RTX 3090 (48G) + DDR5 | Coder-V2 236B | Q4_K_M (135GB) | 3.2 - 4.1 tok/s | Sluggish; PCIe + RAM bandwidth bottleneck |
| Apple Mac Studio Ultra 192G | Coder-V2 236B | Q4_K_M (135GB) | 14.8 - 18.2 tok/s | 100% in Unified Memory (800 GB/s bus) |
| Quad RTX 3090 Cluster (96G) | Coder-V2 236B | IQ2_XXS (78GB) | 28.5 - 34.0 tok/s | Pure 4-way Tensor Parallelism |
Engineering Takeaways:
- The Single GPU Winner: DeepSeek-Coder-V2-Lite (16B) runs flawlessly in pure VRAM on any budget 12GB–16GB GPU. It delivers blistering coding inference inside VS Code without touching system memory.
- The Dual RTX 3090 Reality: If you pair two RTX 3090s (48GB total), Coder-V2 236B requires offloading roughly 65% of its layers to system DDR5 RAM. While pooling system RAM with VRAM avoids OOM crashes, transferring weights over the bus chokes token velocity down to ~3.5 tokens/second. Furthermore, our tests on PCIe x8 vs x16 dual-GPU bandwidth show that PCIe lane saturation introduces noticeable latency spikes during expert swapping.
- Native Linux Advantage: If you are running hybrid offloading or multi-GPU setups, ensure you are running under native Linux rather than Windows DWM to reclaim up to 2GB of trapped VRAM, as shown in our WSL2 vs. Native Ubuntu CUDA benchmarks.
4. Production Launch Blueprint (llama.cpp Hybrid Offload)
When serving DeepSeek-Coder-V2 236B across 48GB VRAM and system memory, balance layer distribution and enable flash attention to prevent driver panics:
bash ./llama-server \ -m deepseek-coder-v2-236b-instruct-q4_k_m.gguf \ --tensor-split 24,24 \ # Distribute evenly across GPU 0 and GPU 1 -ngl 22 \ # Offload exactly 22 compute layers to VRAM -c 32768 \ # 32K Context length --flash-attn \ # Activate FlashAttention-2 kernels --threads 16 \ # Pin to physical CPU performance cores --no-mmap \ # Force contiguous RAM allocation for stability --port 8080  
