How Much VRAM Do You Need to Run an LLM Locally?

By Efe Arda Ergez · Published 21 September 2026

The short answer

Memory needed ≈ weights + KV cache + runtime overhead. For Llama 3.1 8B at the popular Q4_K_M quantisation with an 8K-token context, that is 4.2 GB + 1.0 GB + 0.8 GB = 6.0 GB, within reach of a card with 8 GB. Qwen2.5 32B at the same settings needs 20.0 GB, which fits a 24 GB card — the jump from small to mid-sized models is mostly a jump in memory.

1. The weights

A model is mostly a very large list of numbers. Llama 3.1 8B has 8.03 billion of them. The memory they take is the count multiplied by how many bits each one is stored in:

parameters × bits per weight ÷ 8 = bytes

At full 16-bit precision that is 15.0 GB. Quantisation stores the weights in fewer bits: Q4_K_M averages 4.5 bits per weight, bringing the same model down to 4.2 GB. The quality cost of 4- to 6-bit quantisation is small for most uses; below that it becomes noticeable.

2. The KV cache, which grows with context

While it generates, a model keeps a key and a value vector for every token in the conversation, in every layer. That cache is what lets it “remember” the context, and it grows linearly with the number of tokens:

2 × layers × KV heads × head size × bytes per value × tokens

For Llama 3.1 8B that is 128 KB per token at 16-bit precision. At short contexts the cache is small; at long ones it can outgrow the weights:

• 4K tokens: cache 0.5 GB, total 5.5 GB
• 8K tokens: cache 1.0 GB, total 6.0 GB
• 32K tokens: cache 4.0 GB, total 9.0 GB
• 128K tokens: cache 16.0 GB, total 21.0 GB

Storing the cache in 8 bits instead of 16 halves it, with little measurable quality cost: at 128K tokens the total falls from 21.0 GB to 13.0 GB. Models that use grouped-query attention — most recent ones — have far fewer KV heads than attention heads, which is why their caches are much smaller than older designs of the same size.

3. Runtime overhead

The inference runtime needs working memory of its own for activations, buffers and the CUDA or Metal context. The calculators allow 0.8 GB. Separately, not all of a card’s memory is usable for the model: the display, the driver and other programs take a share, so the calculators count about 92% of a card’s memory as available — 14.7 GB of the 16 GB on an RTX 5070 Ti.

Fitting a bigger model on a smaller card

Qwen2.5 32B at Q4_K_M with an 8K context needs 20.0 GB. On a 24 GB RTX 4090, with 22.1 GB usable, that is a fit; the longest context that fits at those settings is 17K tokens. The levers, in order of how little they cost in quality:

1. Shorten the context, or store the cache in 8 bits.
2. Drop one quantisation step, for example from 5-bit to 4-bit.
3. Offload some layers to system memory — it works, but generation slows sharply.
4. Use a smaller model from the same family.

Mixture-of-experts models are a special case: only a few experts run per token, which makes them fast, but every expert still has to sit in memory. Size them on total parameters, not active ones.

Every model, and the smallest card it fits on

At Q4_K_M, an 8K-token context and a 16-bit cache, smallest first. The card is the one with the least memory, among the consumer cards the calculator covers (10 GB and up), that holds the model with room to spare; each links to what else that card can run. The smallest models also run on cards with less memory than any listed here.

What this leaves out

These are estimates from each model’s published architecture. Real usage varies by runtime and version, by batch size, and by features such as sliding-window attention, which some models use to cap their cache. Memory is also only half the question: a model that fits can still generate slowly if the card’s memory bandwidth is low, and one spread across system memory will be much slower than one held entirely on the card.

Calculators used in this guide

• Local LLM VRAM calculator
• Llama 3.1 8B VRAM requirements
• What LLMs run on an RTX 4090
• What LLMs run on an RTX 5070 Ti

Architectures: Model config.json files published on Hugging Face. Quantisation: GGUF K-quant effective bit widths as produced by llama.cpp. Memory: Manufacturer specifications for VRAM and unified memory. Every figure on this page is computed at build time from the same data and code as those calculators; the formulas are on the Methodology page.

More guides

• How Much Does It Cost to Run a Gaming PC?
• How to Choose a PSU Wattage for Your Build
• How Take-Home Pay Is Calculated in the US