How much VRAM do popular LLMs really need at 4-bit and 8K context? Data from 26 models

Gary Nagra — Mon, 02 Nov 2026 03:59:34 +0000

"Will it fit on my card?" comes up constantly in local-LLM threads, and the usual answer is a rule of thumb like "about half a gigabyte per billion parameters at 4-bit." That rule is close enough for small dense models. For mixture-of-experts models, models with sliding-window attention, and long contexts it can be badly off.

So instead of a rule of thumb, I put together real numbers. For 30 popular open-weight models I recorded the actual GGUF download sizes from Hugging Face and the KV-cache size per token, worked out from each model's config.json. This post uses the 26 models that have both a 4-bit file and a sourced KV-cache figure.

All the data is open (CC BY 4.0) at huggingface.co/datasets/vramneeded/llm-vram-requirements. It's the same data behind the comparison table on vramneeded.com.

The method: weights + KV cache + overhead

The memory a local model uses during inference comes from three parts:

Weights. I use the actual GGUF file size, not params × bits. Real quant files mix precisions: embeddings and some tensors stay at higher precision, and MoE experts in some models stay in MXFP4. The file size captures all of that. Sizes are decimal (1 GB = 10⁹ bytes).
KV cache. For every token in context, each attention layer stores a key and a value vector:

   KV bytes per token = layers × KV heads × head_dim × 2 (K and V) × 2 bytes (FP16)

I count only layers whose cache grows with context. Fixed-size state from sliding-window or linear-attention layers is left out, which is why some newer models have tiny KV numbers. Where I couldn't source the figure, the field is blank. I don't guess.

Overhead. Runtime buffers, the CUDA context, compute scratch and so on. Instead of making up a fixed number, the fit rule simply leaves headroom: a model "fits" a GPU tier only if weights + KV cache come to at most 90% of the card's memory.

The settings used here are the Q4_K_M quant (or the closest 4-bit build where no Q4_K_M exists, noted below), an FP16 KV cache, and an 8,192-token context. The GPU tiers are 8, 12, 16, 24, 32, 40, 48, 64, 96 and 128 GB.

The table

Model	Params	4-bit weights (GB)	KV @ 8K (GB)	Weights + KV (GB)	KV @ 32K (GB)	Smallest tier that fits
LFM3 1.7B	1.7B	1.10	0.10	1.20	0.40	8 GB
Llama 3.2 3B Instruct	3.2B	2.02	0.94	2.96	3.76	8 GB
Spark-X2.6 4B	4.1B	3.54	0.30	3.85	1.21	8 GB
Qwen2.5 7B Instruct	7.6B	4.68	0.47	5.15	1.88	8 GB
Mistral 7B Instruct v0.3	7.2B	4.37	1.07	5.45	4.29	8 GB
Llama 3.1 8B Instruct	8.0B	4.92	1.07	5.99	4.29	8 GB
DeepSeek-R1-0528-Qwen3-8B	8.2B	5.03	1.21	6.24	4.83	8 GB
Qwen3 14B	14.8B	9.00	1.34	10.34	5.37	12 GB
Qwen2.5 14B Instruct	14.8B	8.99	1.61	10.60	6.44	12 GB
gpt-oss-20b	20.9B	11.67	0.20	11.87	0.81	16 GB
LFM2.5-24B-A2B	24.0B	14.53	0.17	14.69	0.67	24 GB
Qwen3.8 27B	27.8B	17.44	0.54	17.98	2.15	24 GB
Qwen3.8-35B-A3B (Empero Distill)	35.1B	21.71	0.17	21.88	0.67	32 GB
Qwen3 32B	32.8B	19.76	2.15	21.91	8.59	32 GB
Qwen2.5 32B Instruct	32.8B	19.85	2.15	22.00	8.59	32 GB
Qwen3.6 35B-A3B	36.0B	22.29	0.17	22.45	0.67	32 GB
Nanbeige4.3 33B	33.1B	20.79	1.74	22.53	6.98	32 GB
Recursive Flint ¹	39.8B	25.05	0.23	25.29	0.94	32 GB
Mixtral 8x7B Instruct v0.1	46.7B	26.44	1.07	27.52	4.29	32 GB
Llama 3.1 70B Instruct	70.6B	42.52	2.68	45.20	10.74	64 GB
Llama 3.3 70B Instruct	70.6B	42.52	2.68	45.20	10.74	64 GB
gpt-oss-120b	116.8B	62.84	0.30	63.14	1.21	96 GB
Ornith-1.5-122B ¹	125.1B	79.71	0.20	79.92	0.81	96 GB
Qwen3.8 Flash Next	180.0B	119.60	0.20	119.80	0.81	over 128 GB
DeepSeek-V4.1-Flash-1017 ¹	763.2B	500.71	0.01	500.71	0.03	over 128 GB
Mistral Large 4	1,049.4B	634.80	0.69	635.49	2.75	over 128 GB

¹ No Q4_K_M build is published, so the 4-bit column is Unsloth's UD-Q4_K_XL.

Per-model notes from the dataset: gpt-oss keeps its MoE experts in MXFP4 in every GGUF, so its Q4_K_M, Q8_0 and BF16 files are close in size. For DeepSeek-V4.1-Flash-1017, keeping the Engram table on SSD brings UD-Q4_K_XL down to 297.9 GB of RAM/VRAM.

The chart: KV cache at 8K context

The cache size per token varies a lot from model to model, and it matters more than people expect. Here is the KV cache at 8K context (FP16) for a selection of the models above:

Llama 3.1/3.3 70B   ███████████████████████████  2.68 GB
Qwen3 32B           ██████████████████████       2.15 GB
Qwen2.5 32B         ██████████████████████       2.15 GB
Nanbeige4.3 33B     █████████████████            1.74 GB
Qwen2.5 14B         ████████████████             1.61 GB
Qwen3 14B           █████████████                1.34 GB
Llama 3.1 8B        ███████████                  1.07 GB
Mixtral 8x7B        ███████████                  1.07 GB
Llama 3.2 3B        █████████                    0.94 GB
Qwen3.8 27B         █████                        0.54 GB
Qwen2.5 7B          █████                        0.47 GB
gpt-oss-120b        ███                          0.30 GB
gpt-oss-20b         ██                           0.20 GB
Qwen3.6 35B-A3B     ██                           0.17 GB

Takeaways

1. At 8K, the KV cache is a small slice for most models, but it grows linearly. Llama 3.1 8B needs 1.07 GB of cache at 8K. At 32K that becomes 4.29 GB, almost as much as its 4.92 GB of weights. If you plan to use long contexts, budget for the cache rather than just the file size.

2. Architecture matters as much as parameter count. Llama 3.2 3B has a 0.94 GB cache at 8K, while the 36B-parameter Qwen3.6 35B-A3B has 0.17 GB, because most of its layers don't keep a growing cache. The 3B model's cache is more than five times bigger. At 32K the gap is 3.76 GB vs 0.67 GB.

3. 8 GB cards cover the whole 7–8B class at 4-bit and 8K. All seven models from 1.7B to 8.2B fit under 90% of 8 GB, with the largest (DeepSeek-R1-0528-Qwen3-8B) at 6.24 GB.

4. 12 GB is the 14B tier and 24 GB tops out around 27B dense. Both 14B Qwen models land at about 10.3–10.6 GB. Qwen3.8 27B comes to 17.98 GB, under the 21.6 GB that 90% of 24 GB allows. The 32B-class models (about 21.9–22.5 GB) go just over that line, so in this rule they need the 32 GB tier. A shorter context or a quantized KV cache can bring them closer, but at 8K with FP16 cache they are over the line.

5. The 4-bit 70B still needs more than one consumer GPU. Llama 3.1/3.3 70B at Q4_K_M plus 8K context comes to 45.20 GB, which is over 90% of 48 GB (43.2 GB). That puts it in the 64 GB tier here.

6. "Half a gigabyte per billion" is a floor, not a forecast. The 4-bit files in this set range from 0.54 GB per billion parameters (gpt-oss-120b: 62.84 GB for 116.8B, with MXFP4 experts) to 0.86 GB per billion (Spark-X2.6 4B: 3.54 GB for 4.1B). Llama 3.1 8B is 0.61 (4.92 GB for 8.0B). Check the real file.

Use the data

Dataset: vramneeded/llm-vram-requirements on Hugging Face. It includes vram-models.csv (one row per model, with source links to the GGUF repo and config.json), quant-sizes.csv (514 quant sizes, not just Q4_K_M) and a cloud GPU price snapshot. Every row has its own checked date.
Calculator and per-model pages: vramneeded.com has the same numbers with a context-length slider and every quant.
Badges: if you publish a model or a model guide, there are free "fits on X GB" README badges at vramneeded.com/badges/. They're built from the same 4-bit, 8K, 90%-headroom rule used in this post.

If a number looks wrong, it may well be: repos get re-uploaded and configs change. Open an issue on the dataset with the source link and I'll fix it.