Mixtral 8x7B VRAM Requirements
46.70B parameters, 32 layers, 8 key-value heads. At Q4_K_M with a 8K context it needs 26.3 GB — 13 of 30 accelerators here can hold it.
A mixture of experts: 8 experts per layer, 2 of them running per token. All of them have to be in memory, so the 46.7B above is what you need to hold — but only 12.88B is doing work at any moment, which is what sets the speed.
Running Mixtral 8x7B
Will not fit on a GeForce RTX 4090
- Weights
- 24.5 GB
- KV cache
- 1.0 GB
- Headroom
- -4.2 GB
- Max context
- 0
Where the memory goes
| Component | Memory |
|---|---|
| Model weights (Q4_K_M) | 24.5 GB |
| KV cache at 8K tokens | 1.0 GB |
| Runtime allowance | 0.8 GB |
| Total | 26.3 GB |
Weights are only half the question
A 46.7B model at Q4_K_M is 24.5 GB of weights. That part is fixed — it is the file on disk, and it does not change while the model runs. What changes is the key-value cache, which grows with every token of context you give it.
Mixtral 8x7B spends 128.0 KB per token. At 8K that is 1.0 GB, comfortably smaller than the weights. At its full 32K it is 4.0 GB — still under the weights, which is unusual for a long-context model. That is the trap: the model loads fine and then fills the card once a conversation gets long.
The reason the cache is as small as it is comes down to grouped-query attention. Mixtral 8x7B has 32 query heads but only 8 key-value heads, and the cache is sized by the latter. Calculators that use the query head count overstate it 4-fold, which is where the wildly pessimistic numbers people quote usually come from. The name is not the size. Eight experts of 7B each come to 46.7B rather than 56B, because the attention layers and embeddings are shared and only the MLPs are replicated. Its config states no separate expert width, so the experts use the model's own intermediate size.
By quantisation
| Format | Bits/weight | Weights | Total at 8K |
|---|---|---|---|
| FP16 / BF16 | 16 | 87.0 GB | 88.8 GB |
| Q8_0 | 8.5 | 46.2 GB | 48.0 GB |
| Q6_K | 6.6 | 35.9 GB | 37.7 GB |
| Q5_K_M | 5.5 | 29.9 GB | 31.7 GB |
| Q4_K_M | 4.5 | 24.5 GB | 26.3 GB |
| Q3_K_M | 3.9 | 21.2 GB | 23.0 GB |
By context length
| Context | KV cache | Total |
|---|---|---|
| 2K tokens | 0.3 GB | 25.5 GB |
| 4K tokens | 0.5 GB | 25.8 GB |
| 8K tokens | 1.0 GB | 26.3 GB |
| 16K tokens | 2.0 GB | 27.3 GB |
| 32K tokens | 4.0 GB | 29.3 GB |
Will it run on your hardware?
| Accelerator | Memory | Runs it | Headroom | Max context |
|---|---|---|---|---|
| GeForce RTX 5090 | 32 GB | Yes | 3.1 GB | 32K |
| GeForce RTX 4090 | 24 GB | No | — | — |
| GeForce RTX 3090 | 24 GB | No | — | — |
| GeForce RTX 5080 | 16 GB | No | — | — |
| GeForce RTX 4080 SUPER | 16 GB | No | — | — |
| GeForce RTX 5070 Ti | 16 GB | No | — | — |
| GeForce RTX 4070 Ti SUPER | 16 GB | No | — | — |
| GeForce RTX 5070 | 12 GB | No | — | — |
| GeForce RTX 4070 | 12 GB | No | — | — |
| GeForce RTX 3060 12GB | 12 GB | No | — | — |
| GeForce RTX 4060 Ti 16GB | 16 GB | No | — | — |
| GeForce RTX 3080 | 10 GB | No | — | — |
| Radeon RX 7900 XTX | 24 GB | No | — | — |
| Radeon RX 7900 XT | 20 GB | No | — | — |
| Radeon RX 9070 XT | 16 GB | No | — | — |
| 2× RTX 3090 (48 GB) | 48 GB | Yes | 17.9 GB | 32K |
| 2× RTX 4090 (48 GB) | 48 GB | Yes | 17.9 GB | 32K |
| 4× RTX 3090 (96 GB) | 96 GB | Yes | 62.0 GB | 32K |
| Mac (M4 Max, 64 GB) | 64 GB | Yes | 21.7 GB | 32K |
| Mac (M4 Max, 128 GB) | 128 GB | Yes | 69.7 GB | 32K |
| Mac (M4 Pro, 48 GB) | 48 GB | Yes | 9.7 GB | 32K |
| Mac (M4, 24 GB) | 24 GB | No | — | — |
| Mac (M3 Max, 128 GB) | 128 GB | Yes | 69.7 GB | 32K |
| Mac Studio (M2 Ultra, 192 GB) | 192 GB | Yes | 117.7 GB | 32K |
| Mac (M1 Max, 32 GB) | 32 GB | No | — | — |
| Mac (M2, 16 GB) | 16 GB | No | — | — |
| NVIDIA A100 40GB | 40 GB | Yes | 11.7 GB | 32K |
| NVIDIA A100 80GB | 80 GB | Yes | 49.7 GB | 32K |
| NVIDIA H100 80GB | 80 GB | Yes | 49.7 GB | 32K |
| NVIDIA L40S 48GB | 48 GB | Yes | 19.3 GB | 32K |
At Q4_K_M with a 8K context. "Only just" means the model uses more than 95% of usable memory — it will load and then fall over as soon as anything else touches the device.
Frequently asked questions
How much VRAM does Mixtral 8x7B need?
What GPU do I need to run Mixtral 8x7B?
Does context length change how much memory Mixtral 8x7B needs?
Why is the cache smaller than other calculators say?
Which quantisation should I use?
Architecture: Hugging Face config.json. GGUF K-quant effective bit widths as produced by llama.cpp.
An estimate of memory, not a benchmark. It counts model weights, the key-value cache at the context you choose, and a fixed runtime allowance. Actual usage moves with the runtime, the batch size, whether flash attention is on, and how much the operating system has already taken from a shared memory pool. Treat a result within a gigabyte of your card's capacity as 'probably not' rather than 'just fits'.
Data on this page last verified .