Calculate the GPU memory an LLM needs. Llama, Gemma, Qwen, DeepSeek or any Hugging Face model, at every quantization and context size, plus which GPUs fit.
Pick a model, pick a quantization, set the context length, and this calculator gives you three numbers — weights, KV cache, runtime overhead — and the total. Underneath it checks that total against a list of consumer GPUs, workstation and datacenter cards, AMD and Intel parts, Apple Silicon and CPU-only configurations, and tells you for each one whether the model fits, fits only at a reduced context, needs multiple cards, or will not run.
This is an inference calculator: memory to run a model for generation. Training and LoRA fine-tuning add optimizer state and gradients and are a different arithmetic entirely — that is what the separate fine-tuning VRAM calculator is for. Everything computes in your browser from a bundled dataset; the only network request the tool ever makes is the optional Hugging Face lookup described below.
No hand-waving — here is the exact calculation:
total = weights + kv_cache + overhead
parameter_count × bytes_per_parameter. The bytes-per-parameter figures are calibrated against real GGUF file sizes, not the nominal bit width, because embedding and output layers are usually stored at higher precision than the body of the model. Q4_K_M is therefore 0.61 bytes per parameter, not 0.5.2 × kv_heads × head_dim × kv_bytes × (global_layers × context + local_layers × min(context, sliding_window)). The leading 2 is the K and the V. Layers using sliding-window attention cap their share of the cache at the window size instead of growing with the full context.layers × mla_kv_dim × context × kv_bytes, because MLA stores one compressed latent vector per token per layer rather than a K and V per head.max(6% of weights, 0.75 GiB). This stands in for the CUDA or Metal context, activation buffers and allocator fragmentation. The floor matters for small models: a 1B model does not get away with 60 MB of overhead.All results are in binary gigabytes (GiB, 10243 bytes), which is how GPU memory is actually addressed and how inference engines report it.
Two things this formula deliberately does not model, and you should know it: batch size (the KV cache figure is for a single sequence — a server handling eight concurrent requests needs roughly eight times the cache) and CPU offload. The total is what a full GPU-resident deployment costs.
The parameter count is a count of numbers, not of bytes. What it costs depends entirely on how many bytes you spend storing each one. Llama 3.1 8B, which really has 8,030,261,248 parameters, at 8,192 tokens of context with an FP16 cache:
| Quantization | Bytes/param | Weights | KV cache | Overhead | Total |
|---|---|---|---|---|---|
| FP16 / BF16 | 2.00 | 14.96 GB | 1.00 GB | 0.90 GB | 16.85 GB |
| Q8_0 | 1.06 | 7.93 GB | 1.00 GB | 0.75 GB | 9.68 GB |
| Q6_K | 0.82 | 6.13 GB | 1.00 GB | 0.75 GB | 7.88 GB |
| Q4_K_M | 0.61 | 4.56 GB | 1.00 GB | 0.75 GB | 6.31 GB |
| Q3_K_M | 0.50 | 3.74 GB | 1.00 GB | 0.75 GB | 5.49 GB |
So the same model ranges from not fitting on a 16 GB card to fitting comfortably on an 8 GB one, purely on the quantization choice. The naive "8B means 8 GB" heuristic happens to land near the Q8 answer by coincidence and is wrong in both directions everywhere else.
On quality: Q8 is described in the tool as practically indistinguishable from FP16, Q6_K as near-lossless, and Q4_K_M as the popular balance — which is why Q4_K_M is the default. Below that, Q3_K_M is flagged as visibly degraded and IQ2_M as a last resort for models that otherwise will not load at all. The practical rule most people converge on: a larger model at Q4 beats a smaller model at FP16 for the same memory budget, until you get down to 3-bit and below, where the degradation starts to cost you more than the extra parameters buy.
The weights are a fixed cost. The KV cache is a per-token cost that grows linearly with how much context you keep in play, and it is the thing that turns a model that "fits" into one that crashes forty minutes into a long document.
Same Llama 3.1 8B at Q4_K_M, FP16 cache, varying only the context:
| Context | Weights | KV cache | Total | Consequence |
|---|---|---|---|---|
| 4K | 4.56 GB | 0.50 GB | 5.81 GB | Fits an 8 GB card with room to spare |
| 8K | 4.56 GB | 1.00 GB | 6.31 GB | Still fits 8 GB |
| 32K | 4.56 GB | 4.00 GB | 9.31 GB | Needs 10 GB or more |
| 128K | 4.56 GB | 16.00 GB | 21.31 GB | Cache is now bigger than the model — 24 GB card territory |
At the model's full 128K window the cache is more than three times the size of the weights. This is the single most common surprise for people sizing hardware, and it is why the tool has a context slider at all rather than a fixed assumption.
The mitigation is the KV cache precision toggle. Switching from FP16 to Q8 halves every cache figure above at very little quality cost — that same 8B at 128K drops from 21.31 GB to 13.31 GB, the difference between needing a 24 GB card and running on a 16 GB one. In llama.cpp this is --cache-type-k q8_0 (pair it with --cache-type-v q8_0).
Architecture matters here too, and the calculator accounts for it rather than assuming. Grouped-query attention is why a modern 8B with 8 KV heads has a far smaller cache than an older model with 32; sliding-window layers cap their contribution at the window size, so a Gemma-style model with mostly local layers and a small window barely grows its cache with context at all; and MLA models compress the cache to a single latent per token per layer. Two models with identical parameter counts can differ by an order of magnitude in cache size.
Below the breakdown, every piece of hardware in the dataset is checked against your total and grouped by vendor and class. Four possible verdicts:
--ctx-size to.One important detail in how the table treats unified memory: for Apple Silicon and CPU configurations, only 75% of total system memory is counted as usable. That memory is shared with the operating system, the window server and everything else you have open, and macOS will not let a single process claim all of it. A 64 GB Mac is treated as a 48 GB budget. Discrete GPUs are counted at their full VRAM, since the display driver's own consumption is small relative to the card and is partly what the overhead term covers.
Llama 3.3 70B (70,553,706,496 parameters, 80 layers, 8 KV heads, head dim 128) at Q4_K_M with an 8K FP16 cache:
What that means across hardware: a single RTX 4090 (24 GB) cannot hold it at any context, because the weights alone exceed the card — the table reports 2 GPUs needed. A single 48 GB workstation card fits it at 8K with about 94% utilisation, which is tight enough that raising the context will fail. An M4 Max in its larger memory configuration has a 72 GB usable budget and fits the same setup at roughly 62%, with headroom to push the context up. Raise the context to 32K and the total climbs to 52.49 GB, which puts the 48 GB card out and leaves the Mac and the 80 GB datacenter parts in.
This is the shape of the whole exercise: the weights decide whether a class of hardware is even in the conversation, and the context decides where inside that class you land.
The calculator tells you whether you fit. It does not model what happens when you do not, so here is the honest picture.
With llama.cpp, Ollama or LM Studio, exceeding VRAM is not usually a crash. These engines let you place only some layers on the GPU (-ngl / the GPU-layers setting) and keep the rest in system RAM, executing those on the CPU. The model runs. It runs slowly, and the slowdown is not proportional to the fraction offloaded — it is dominated by the fact that token generation is memory-bandwidth-bound and system RAM has a small fraction of the bandwidth of GPU memory. Offloading even a modest share of layers can cost most of your throughput, and the transition is sharp rather than gradual. Prompt processing degrades too, on top of generation.
With vLLM, TensorRT-LLM or ExLlama, there is generally no graceful degradation: you get a CUDA out-of-memory error at load, or the server refuses to start because it cannot reserve the KV cache blocks you configured. That is arguably the better behaviour, because it fails at startup rather than at 2 a.m. under load.
The two failure modes to plan around are therefore different. If the weights do not fit, you are choosing between offload (slow), a smaller quantization (lower quality), a smaller model, or more cards. If only the cache pushes you over, you have cheaper options first: quantize the cache to Q8, reduce the context, or reduce concurrency.
Leave headroom regardless. A card sitting at 99% in the table will out-of-memory the first time a long prompt arrives, a second process claims memory, or the allocator fragments after hours of varied request sizes. Treating roughly 85–90% as your practical ceiling costs you very little and prevents the class of failure that is hardest to reproduce.
The picker has three modes. Popular is a curated list of open-weight models whose architecture figures — layers, KV heads, head dimension, native context, attention type — were taken from each model's config.json, with parameter counts from safetensors metadata. Search Hugging Face queries the Hub live for anything not on the list; if the repository does not publish a parameter count the tool will say so rather than guess, and quantized re-uploads frequently do not, so the original repository is the one to search for. Custom lets you type the numbers directly — parameters, layers, KV heads, head dimension, max context, and active parameters if it is a mixture-of-experts model.
On MoE: the calculator sizes memory from total parameters, which is correct. All experts must be resident in memory even though only a fraction are read per token. Active parameter count affects speed, not footprint, which is why the tool passes your current model, quantization and context through to the companion inference speed calculator rather than trying to answer both questions on one screen.
Your selections are encoded in the URL as you change them, so a configuration can be shared or bookmarked and comes back exactly as you left it. Switching to a model with a smaller native context automatically clamps the slider rather than silently computing an impossible setup.
Three things consume GPU memory when running a language model:
1. Model weights — the parameters themselves. This is the big one: parameter count x bytes per parameter. A 8B model at FP16 is 16 GB; the same model quantized to Q4_K_M is about 4.9 GB.
2. KV cache — the attention state for every token in your context window. This grows linearly with context length and depends on the model's architecture: number of layers, KV heads, and head dimension. Modern architectural tricks (grouped-query attention, sliding-window layers, DeepSeek's MLA compression) exist mostly to shrink this number.
3. Runtime overhead — CUDA or Metal context, activation buffers, and memory fragmentation. Typically 5-10% of the model size, with a floor of roughly 0.5-1 GB.
The calculator above computes all three from each model's real architecture (pulled from its config.json on Hugging Face), so the totals match what you'll actually see in nvidia-smi or Activity Monitor.
Quantization is the single most effective way to fit a bigger model on your hardware. It stores weights at lower numeric precision:
| Format | Bytes/weight | Quality impact |
|---|---|---|
| FP16 | 2.0 | Reference quality |
| Q8_0 | ~1.06 | Indistinguishable from FP16 |
| Q6_K | ~0.82 | Near-lossless |
| Q5_K_M | ~0.71 | Very good |
| Q4_K_M | ~0.61 | Good — the popular default |
| Q3_K_M | ~0.50 | Noticeable degradation |
| IQ2_M | ~0.36 | Significant degradation |
The practical guidance: Q4_K_M is the sweet spot for most use. Larger models tolerate aggressive quantization better than small ones — a 70B model at Q3 usually beats a 14B model at Q8. If you have spare VRAM, step up to Q5_K_M or Q6_K rather than leaving it idle.
The KV cache can be quantized too (llama.cpp's cache-type-k q8_0 flag), which halves context memory at negligible cost — worth doing for long-context work.
When one GPU is not enough, you have two fundamentally different paths:
Multiple NVIDIA/AMD GPUs. VRAM stacks: two 24 GB cards hold a ~45 GB model. Layer splitting (llama.cpp, Ollama) is the easy path — it works with mismatched cards and normal PCIe slots, but speed stays at single-GPU levels. Tensor parallelism (vLLM, TensorRT-LLM) actually multiplies bandwidth, but wants identical GPUs and fast interconnect. Used 3090s remain the budget favorite for this: 24 GB and 936 GB/s per card.
Apple Silicon unified memory. A Mac Studio with 128-512 GB of unified memory can hold models that would need 4-8 discrete GPUs. Bandwidth is the trade-off: 546 GB/s (M4 Max) to 819 GB/s (M3 Ultra) versus 1000-3350 GB/s for discrete cards, so generation is slower. For models above ~50 GB, Apple Silicon is often the cheapest hardware that runs them at all.
One thing that does NOT work: combining VRAM across machines over a network, or mixing Apple unified memory with discrete GPUs. The memory pool must be on one machine.
As a rule of thumb at the popular Q4_K_M quantization: 7-9B models need about 6-8 GB, 13-14B models need about 10-12 GB, 27-32B models need about 20-24 GB, and 70B models need about 43-48 GB (two 24 GB GPUs). Add more for long context windows — the KV cache grows linearly with context. This calculator computes the exact number for any model, quantization, and context size.
Total VRAM = model weights + KV cache + runtime overhead. Weights = parameter count x bytes per parameter (2 bytes at FP16, ~0.6 bytes at Q4_K_M). KV cache = 2 x layers x KV heads x head dimension x context length x bytes per element — this is what grows when you increase context. Overhead (~6%, minimum ~0.75 GB) covers CUDA/Metal buffers, activations, and memory fragmentation.
Quantization stores model weights at lower precision to save memory. Compared to FP16 (2 bytes per weight): Q8_0 halves memory with virtually no quality loss, Q4_K_M cuts it to about 30% with minor quality loss (the most popular choice), and 2-bit quants cut it to about 18% with significant quality loss. A 70B model goes from 141 GB at FP16 to about 43 GB at Q4_K_M — the difference between needing a server rack and two consumer GPUs.
The KV cache stores the attention keys and values for every token in your context window, so the model does not have to recompute them for each new token. It grows linearly with context length — at 8K context it is usually small, but at 128K it can exceed the size of the model weights themselves. Models with grouped-query attention (most modern LLMs), sliding-window layers (Gemma), or MLA compression (DeepSeek) need dramatically less KV cache than older architectures.
Not entirely on the GPU at useful quality. A 70B model at Q4_K_M needs about 43 GB. Your options: 1) Use two 24 GB GPUs (llama.cpp and vLLM split models across GPUs automatically). 2) Run a smaller quant like IQ2_M (~25 GB) — quality suffers noticeably. 3) Offload some layers to system RAM — works but generation slows to a crawl (often under 2 tokens/sec). 4) Use the 70B model's smaller siblings (Qwen3 32B and Gemma 3 27B at Q4 fit in 24 GB and are surprisingly capable).
Yes — VRAM capacity stacks across GPUs. Layer split (llama.cpp/Ollama default) puts different layers on different GPUs; it works with mismatched cards over PCIe but does not increase speed. Tensor parallel (vLLM, TensorRT-LLM) splits every layer across identical GPUs and scales bandwidth almost linearly, but needs fast interconnect. Apple Silicon cannot combine memory across machines — the unified memory ceiling is the limit.
Yes, very well. Apple Silicon shares one memory pool between CPU and GPU, so a 128 GB M4 Max can hold models that would need multiple NVIDIA GPUs. By default macOS lets the GPU use about 75% of unified memory (this calculator accounts for that). The trade-off is bandwidth: M4 Max (546 GB/s) is roughly half an RTX 4090 (1008 GB/s), so generation is slower — but for large models that do not fit in 24 GB of VRAM at all, slow beats impossible.
Inference engines fall back to CPU offloading — some layers run on the GPU, the rest on the CPU from system RAM. It works, but every offloaded layer is bottlenecked by system RAM bandwidth (50-90 GB/s vs 1000+ GB/s on a GPU), so speed drops sharply. A model that is 20% offloaded can be 3-5x slower than one that fits entirely. If you see single-digit tokens/sec, this is usually why.
Mixture-of-Experts models like Qwen3 30B-A3B only read about 3B parameters per token, which makes them fast. But all 30B parameters must still be loaded in memory, because different tokens route to different experts and the engine cannot predict which. So MoE models need the VRAM of their total size but generate at the speed of their active size — the best of one world, the cost of the other.
Within about 5-10% for typical setups. Architecture specs (layers, KV heads, head dimensions) come from each model's actual config.json on Hugging Face, and quantization sizes match real GGUF file sizes. Real usage varies by inference engine: llama.cpp, vLLM, ExLlama, and MLX each have different overhead, context pre-allocation, and padding behavior. Treat results as a close estimate for planning hardware, not a guarantee.
Estimate LLM tokens per second from memory bandwidth, model size, quantization, and context window. Compare generation speed across GPUs and understand the memory-bandwidth bottleneck.
Calculate AWS Bedrock costs for Claude, Llama, Titan, and other AI models. Estimate input/output token costs for your workload
Compare cloud computing costs across AWS, Azure, and Oracle Cloud. Get instant pricing estimates for compute instances with real-time data.
Count tokens in text for GPT-4, Claude, Llama, and other LLMs. Estimate API costs and optimize prompts for token limits
Detect your GPU with one click and see which LLMs your computer can actually run — ranked by whether they fit in your VRAM, need CPU offloading, or will not run at all.
Calculate VRAM requirements for fine-tuning LLMs with full fine-tuning, LoRA, or QLoRA — accounting for gradients, optimizer states, activations, and gradient checkpointing.