Estimate LLM tokens per second on your GPU from memory bandwidth, model size, quantization and context. Covers RTX 4090, 5090, A100, H100, Apple Silicon.
Pick a model, a quantisation, a GPU and a context length, and this calculator projects the two speeds that decide how an LLM feels to use: how fast it generates tokens, and how long you wait before the first one appears. It does not measure your machine and it does not download anything — it is arithmetic over published hardware specs and each model's real architecture, so you can answer "will a 3060 handle this" before buying, downloading, or renting anything.
For a single user generating one token at a time, an LLM is not compute-limited. Every new token requires reading the whole set of active weights plus the entire KV cache out of memory, then doing a comparatively trivial amount of arithmetic on them. The GPU finishes the maths long before the memory can deliver the next batch of bytes. So:
tokens/sec ≈ memory bandwidth ÷ bytes read per token
The calculator applies that formula at 50–75% of theoretical bandwidth and shows the result as a range rather than a single number, because real utilisation depends on which engine you run — llama.cpp, vLLM and MLX all land in different parts of that band. This is why TFLOPS is close to the wrong number to shop on. An RTX 4090 has 83 TFLOPS of FP16 and 1,008 GB/s; an RTX 5090 has 105 TFLOPS and 1,792 GB/s. The 27% more compute is nearly irrelevant to single-stream chat; the 78% more bandwidth is the entire story.
Llama 3.1 8B at Q4_K_M with 8,192 tokens of context reads 4.56 GiB of weights and 1.00 GiB of KV cache per token — 5.56 GiB in total. Divide the bandwidth of each card by that figure:
| Hardware | Bandwidth | Generation | Prompt processing | Time to first token (1K prompt) | 500-token reply |
|---|---|---|---|---|---|
| RTX 3060 | 360 GB/s | 30–45 tok/s | ~405 tok/s | ~2.5 s | ~15.8 s |
| Apple M3 Max | 400 GB/s | 33–50 tok/s | ~872 tok/s | ~1.2 s | ~13.1 s |
| RTX 4090 | 1,008 GB/s | 84–127 tok/s | ~2,584 tok/s | ~0.4 s | ~5.1 s |
| RTX 5090 | 1,792 GB/s | 150–225 tok/s | ~3,269 tok/s | ~0.3 s | ~3.0 s |
Notice that the M3 Max and the 3060 generate at almost the same rate — their bandwidths are within 11% of each other — while their prompt-processing rates differ by more than two to one. That is the second half of the model, and it is a different bottleneck entirely.
Prompt processing (prefill) reads your entire prompt at once and can therefore run every token through the network in parallel. That makes it compute-bound, and the calculator models it as roughly two floating-point operations per active parameter per token at about 50% of the card's FP16 throughput. Decode — generating the reply one token at a time — cannot be parallelised the same way and stays memory-bound. The practical consequences:
The prompt-length slider exists to make that visible: it changes time to first token and the total time for a 500-token reply, and leaves the generation rate alone.
The KV cache is read on every single token, so as a conversation grows the per-token byte count grows with it and generation slows down. For the same Llama 3.1 8B at Q4_K_M on an RTX 4090:
| Context in use | KV cache read per token | Share of memory traffic | Estimated generation |
|---|---|---|---|
| 4,096 | 0.50 GiB | 10% | ~116 tok/s |
| 8,192 | 1.00 GiB | 18% | ~105 tok/s |
| 32,768 | 4.00 GiB | 47% | ~69 tok/s |
| 131,072 | 16.00 GiB | 78% | ~29 tok/s |
At full 128K context the cache, not the weights, is most of what the GPU reads — the model is spending three quarters of its memory bandwidth on the conversation history. The tool flags this once the KV share passes 30% and tells you what the remedies are: run a shorter context, or quantise the KV cache. That is not a hypothetical: it is the reason a chat that started snappy feels sluggish forty messages in.
The size of the cache depends on architecture, not just on model size. Models with sliding-window attention cap most of their layers at the window rather than the full context, and DeepSeek-style multi-head latent attention stores one compressed vector per token per layer instead of full key and value tensors. The calculator uses each model's real layer counts, KV head counts and head dimensions, so those savings show up rather than being averaged away.
Because speed is bytes divided by bandwidth, cutting the bytes cuts the time proportionally. Moving Q8_0 (1.06 bytes per parameter) to Q4_K_M (0.61) shrinks the weight traffic by roughly 42% and speeds up generation by about the same, which is why 4-bit is the default choice for local use even on cards with memory to spare.
Mixture-of-experts models break the usual relationship between size and speed, because only the routed experts are read per token. Qwen3 30B-A3B stores 30.5 billion parameters but touches about 3.3 billion per token, so on an RTX 4090 it projects to 179–268 tok/s — faster than a dense 8B on the same card, despite being nearly four times the download. GPT-OSS 120B reads about 5.1 billion of its 120 billion. The calculator uses active parameters for speed and total parameters for capacity, which is exactly the asymmetry that makes MoE attractive if you have the memory to hold it.
Tokens per second is an abstract unit until you have sat in front of it. The tool labels its own output on this scale, and the boundaries are worth internalising when you are deciding whether a configuration is good enough:
| Generation speed | How it reads |
|---|---|
| 50 tok/s and above | Very fast — output appears essentially instantly |
| 20–50 tok/s | Fast — comfortably ahead of reading speed |
| 10–20 tok/s | Comfortable — roughly reading speed |
| 4–10 tok/s | Usable but slow; fine for background jobs, tiring for chat |
| Below 4 tok/s | Painfully slow |
Two caveats on reading that scale. Reasoning models emit long chains of thought before the visible answer, so a nominally comfortable 12 tok/s can still mean a thirty-second wait for a short reply. And code generation is consumed differently from prose — nobody reads a generated file line by line at reading speed, so the same rate feels slower when the output is a diff.
Adding a second card does not automatically double anything, and the tool makes you say which mode you are running:
This is the single most common source of disappointment in multi-GPU builds: people buy a second card expecting throughput and get headroom.
Models come from a curated list of popular open weights, from a live Hugging Face search that pulls real architecture from the repository's configuration, or from a custom entry if you want to model something unreleased. Hardware comes from the built-in table — NVIDIA consumer through B200, AMD including MI300X, Intel Arc, every Apple Silicon tier, and plain CPU + DDR4/DDR5 — or from a custom option where you type a bandwidth and TFLOPS figure directly. Below the results, a bar chart ranks every card from one vendor for your exact model, quantisation and context, and marks which of them the model actually fits on, so the fastest option and the feasible option are visible in the same place. The full configuration is encoded in the URL for sharing.
The most counterintuitive fact about running LLMs: the GPU's math performance (TFLOPS) barely matters. What matters is memory bandwidth.
Here is why. To generate one token, the model must read every active parameter from memory exactly once, and perform only about 2 floating-point operations per parameter read. Modern GPUs can execute hundreds of operations in the time it takes to fetch one value from VRAM. The compute units spend most of their time waiting.
So the speed formula is simple: tokens/sec = memory bandwidth / bytes per token, derated by 50-75% for real-world engine efficiency.
This explains otherwise-strange benchmark results: an RTX 3090 (936 GB/s) nearly matches an RTX 4080 (717 GB/s) despite being far behind on paper compute. It is also why datacenter GPUs cost what they do — an H100's 3350 GB/s of HBM bandwidth is its real product, not its TFLOPS.
LLM inference has two phases with completely different performance characteristics:
Prefill (prompt processing) happens once, before the first token appears. All prompt tokens are processed in parallel, which saturates the GPU's compute units — this phase IS compute-bound. Time to first token = prompt length / prefill speed. This is why pasting a long document into a local model causes a long pause before anything happens.
Decode (generation) produces tokens one at a time, each requiring a full read of the active weights and KV cache — memory-bound, as described above.
The practical consequences: a GPU with strong compute but modest bandwidth (like an RTX 4060 Ti) will have snappy prompt processing but mediocre generation speed. Apple Silicon is the opposite — generation speed is respectable for its bandwidth, but prefill on long prompts is slow because its GPU compute is far below discrete cards. If your workload involves repeatedly feeding long documents, prefill speed may matter more to you than tokens/sec.
Ranked by impact:
1. Make sure the model fully fits in VRAM. Partial CPU offloading is the number one cause of disappointing speed — even 10% of layers on the CPU can halve your throughput. Check with the VRAM calculator first.
2. Use a smaller quantization. Q4_K_M reads ~70% less data per token than FP16 — that is a direct ~3x speedup. Quality loss is modest; this is the best speed-per-quality trade available.
3. Pick a MoE model. Qwen3 30B-A3B reads ~3B parameters per token; Qwen3 32B (dense) reads all 32B. The MoE model generates roughly 8-10x faster on identical hardware with broadly comparable capability.
4. Buy bandwidth, not compute. When choosing hardware for LLMs, sort by memory bandwidth per dollar. Used RTX 3090s (936 GB/s) routinely beat newer, more expensive cards.
5. Keep context lean. Past ~32K tokens, the KV cache meaningfully adds to per-token reads. Summarize or truncate long conversations, or use KV cache quantization.
6. Use tensor parallelism for multi-GPU. If you have matched GPUs, vLLM with tensor parallel scales speed; llama.cpp's default layer split does not.
For interactive chat: 20+ tokens/sec feels instant (faster than you can read), 10-20 tokens/sec is comfortable, 5-10 tokens/sec is usable but noticeably slow, and below 5 tokens/sec gets frustrating. For agentic workloads (coding assistants, tool use) where the model generates long outputs you skim rather than read, higher speeds matter much more — 50+ tokens/sec is the difference between a 30-second and a 3-minute task.
Single-user generation is memory-bound: every new token requires reading all active model weights plus the KV cache from memory. So tokens/sec = effective memory bandwidth / bytes read per token. A 8B model at Q4_K_M is about 5 GB of reads per token; on an RTX 4090 (1008 GB/s at 50-75% real-world efficiency) that gives roughly 85-125 tokens/sec. GPU compute (TFLOPS) barely matters for this — bandwidth is everything.
Because generating one token requires reading every active weight from memory but only doing 2 floating-point operations per weight read. Modern GPUs can do hundreds of operations in the time it takes to read one value from memory — so the math units sit idle waiting for data. This is why an RTX 3090 (936 GB/s, 36 TFLOPS) generates tokens almost as fast as an RTX 4080 (717 GB/s, 49 TFLOPS) despite being two generations older: it has more bandwidth.
Mixture-of-Experts models only route each token through a few experts, so they read a fraction of their parameters per token. GPT-OSS 120B reads only ~5.1B parameters per token — so it generates about as fast as a 5B dense model while having 120B-model knowledge. This is why MoE has taken over: Qwen3 30B-A3B, Gemma 4 26B-A4B, and DeepSeek V4 all generate at small-model speeds. The catch: you still need enough memory to hold all the parameters.
TTFT is the delay between sending your prompt and seeing the first word of the response. It is dominated by prefill — processing your prompt — which unlike generation is compute-bound (all prompt tokens process in parallel). Long prompts on weak hardware mean long waits: a 32K-token prompt on Apple Silicon can take 30+ seconds before anything appears, while the same prompt on an H100 takes about a second. This calculator estimates both TTFT and generation speed.
Only with tensor parallelism (vLLM, TensorRT-LLM), which splits every layer across GPUs and scales bandwidth to roughly 90% per added card — so 2 GPUs give about 1.9x speed. The default in llama.cpp and Ollama is layer splitting, which adds VRAM capacity but NOT speed: GPUs process their layers in turn, so you get one GPU's worth of throughput. If you have two identical GPUs and want speed, use vLLM with tensor parallel.
Common reasons: 1) Layers offloaded to CPU because the model does not quite fit in VRAM (the biggest one — check with the VRAM calculator). 2) The inference engine is not optimized for your hardware (llama.cpp Metal vs CUDA vs ROCm differ a lot). 3) Long context — the KV cache adds reads per token as the conversation grows. 4) Power/thermal limits, especially on laptops. 5) Batch size 1 is assumed here; serving multiple users splits bandwidth between them.
Every generated token must read the entire KV cache (the stored attention state for all previous tokens) in addition to the model weights. At short contexts this is negligible, but at 64K+ tokens of conversation, the KV cache can add gigabytes of reads per token and cut generation speed by 30-50%. Models with MLA (DeepSeek), sliding-window attention (Gemma), or aggressive grouped-query attention degrade much less with long context.
Apple Silicon trades speed for capacity. An M4 Max (546 GB/s) generates at roughly half the speed of an RTX 4090 (1008 GB/s) for models that fit on both — think 40-60 tokens/sec for an 8B Q4 model vs 85-125 on the 4090. But the M4 Max can have 128 GB of unified memory, letting it run 70B+ models that need two or three consumer GPUs. For large models, a Mac is often the cheapest way to run them at all; for small models, a discrete GPU is much faster.
Calculate how much VRAM any LLM needs to run locally. Pick a model, quantization, and context size — see download size, total memory required, and which GPUs it fits on.
Calculate AWS Bedrock costs for Claude, Llama, Titan, and other AI models. Estimate input/output token costs for your workload
Model API rate limits, concurrency, and burst behavior to build throttling that avoids 429 errors. Get per-client budgets, queue sizing tips, and token bucket parameters instantly.
Count tokens in text for GPT-4, Claude, Llama, and other LLMs. Estimate API costs and optimize prompts for token limits
Speed test your GPU for AI — measure real memory bandwidth and compute with WebGPU, run an actual LLM in your browser, and see predicted speeds for every popular model on your hardware.
Detect your GPU with one click and see which LLMs your computer can actually run — ranked by whether they fit in your VRAM, need CPU offloading, or will not run at all.