Tokens per Second Benchmarks Across Consumer GPUs

Consumer GPU tokens-per-second benchmarks are best read as tests of memory bandwidth. For single-user token generation, the key question is how quickly a card can read the model’s weights, provided they fit in memory. That gives GPU selection a clear order. Establish model fit first, then compare bandwidth under conditions that match the intended workload.

Why tokens per second is not what most benchmark tables make it look like

Two sources can give very different tokens-per-second figures for the same GPU, and both numbers can be right. What changed between the tests? One may use llama.cpp while another uses Ollama or vLLM, with different quantization, context length, or batch size. The discrepancy is a solvable problem, but the number alone cannot solve it. Tokens per second is the right headline metric for local inference only when those conditions travel with it.

There is another source of confusion in the compute specifications. A high-end data-center GPU can have theoretical arithmetic throughput for a small model many orders of magnitude above what production inference engines deliver. That gap is not simply a software failure. It reflects a workload that cannot keep the arithmetic units busy. The specification that looks most impressive may not explain the result.

Why token generation is memory-bound, not compute-bound

Decoder-only LLM inference has two phases, which are prefill and decode. Prefill processes the prompt, while decode generates the answer one token at a time. Each new token depends on the previous step. For a dense model, that loop requires a full read of the model’s weights from VRAM on every decoding step, so the memory bus limits how quickly generation can proceed. More CUDA cores do not remove that sequential dependency.

FastTPS, an arXiv 2026 paper, identifies the token phase’s sequential nature as a fundamental throughput constraint on AI accelerators. It also says that frequent KV cache load-store operations use more of the memory bandwidth. At batch size 1, arithmetic intensity is approximately 1 FLOP per byte, leaving little computation for each byte moved. The arithmetic units finish their work quickly and then have to wait. Adding more TFLOPS does little until that memory bottleneck changes. A chip with lower peak compute can therefore match or beat one with greater arithmetic throughput.

How memory bandwidth predicts the benchmark table before you look at it

The RTX 3090 makes the bandwidth argument concrete. Its memory bandwidth is close to the RTX 4090’s, so its single-user token generation speed is also close, while its used-market price is a fraction of the newer card’s price. A TFLOPS comparison favoring a newer mid-range card can therefore point a buyer in the wrong direction for this workload. With the model and test conditions held constant, bandwidth per dollar is the more useful comparison.

The RTX 5060 Ti 16GB illustrates the reverse. Its roughly 448 GB/s of memory bandwidth is less than half the 4090’s, which predicts substantially lower throughput on the same models. AMD’s RX 7900 XTX reaches a large fraction of the 4090’s tokens per second on comparable workloads, consistent with its relative bandwidth position. The pattern extends beyond one chip maker. A benchmark table is, to a useful first approximation, a bandwidth table in disguise, though software determines how much of that bandwidth becomes useful token generation.

VRAM capacity as the second constraint when bandwidth becomes irrelevant

Bandwidth helps only while the workload fits in VRAM. If the model’s weights take up more space than the card has, CPU offload moves some of the work across a much slower bus. The discrete GPU’s bandwidth advantage then breaks down. A fast card with too little memory is not simply a slightly slower version of a suitable card. It has crossed into a different operating condition, where its local memory specification no longer describes the whole job.

Unified memory offers a structural answer to that problem. Apple Silicon and AMD Strix Halo allow the CPU and GPU to share a large memory pool, so very large models can stay entirely on-device at usable speeds. A 24 GB NVIDIA card may fail to run the same model, or it may run more slowly after CPU offload. The RTX PRO 6000 Blackwell approaches the problem from the discrete-card side with 96 GB of VRAM, extending model fit beyond the consumer tier while remaining subject to the capacity constraint.

Quantization as the lever that adjusts both constraints simultaneously

Quantization changes both sides of the decision at once. With fewer bytes per weight, the model takes up less VRAM and moves less data through memory for each token it generates. At batch size 1, moving from FP16 to Q4 approximately quarters the bytes per weight and could quadruple decode speed under ideal bandwidth-bound conditions. Real gains are typically much smaller. Kernel overhead and dequantization costs prevent the reduction in stored bytes from translating directly into an equal speedup.

Does that simply buy speed by making the model worse? Q4_K_M on a modern mid-to-large model typically outperforms an unquantized smaller model in quality benchmarks, so the relevant comparison is not always the same model before and after compression. The choice may be between two different model sizes that fit the same card. Practitioners adjust the dial to their memory budget, with quality becoming less favorable as compression grows more aggressive.

Q2 shows how far that dial can move. It makes some very large models runnable on 24 GB cards, but at the cost of significant quality degradation. That explains why consumer GPU tables may show large-model results only at Q2. Tiny quantized models occupy the other end of the spectrum, achieving throughput far above mid-sized models because each token requires so few weight bytes. Neither result is a property of the GPU alone, so comparisons across quantization levels or model sizes answer different questions.

Context length and batch size change the benchmark answer

The throughput figures that get top billing are usually for short-context, single-user generation. As the active context grows, the KV cache stores more attention state and consumes a larger share of both VRAM capacity and bandwidth. During decode, the model reads these along with its weights. Moving from a short to a very long context can produce a roughly 40% drop in tokens per second on the same card and model. A benchmark near the start of a conversation therefore need not describe performance deep into it.

Concurrent users change the workload in a different direction. At higher batch sizes, inference shifts toward being compute-bound rather than purely bandwidth-bound, giving stronger arithmetic throughput a larger role. This is where data-center GPUs’ compute advantages become relevant. A solo developer running a local chatbot and someone serving a small team are not necessarily choosing hardware for the same regime. Check context length and batch size, as well as model, quantization, and backend, before treating a published result as a prediction.

GPU-by-GPU breakdown through the bandwidth framework

The RTX 5090 combines 32 GB of GDDR7 with the highest consumer memory bandwidth available on a discrete NVIDIA card. That bandwidth makes it the fastest at single-GPU token generation for models that fit in its memory. Its extra memory also broadens model fit. It can handle sizeable models comfortably and quantized very-large models in a pinch, but not a very large model at high precision. Its Blackwell FP4 support remains incompletely exploited by mainstream inference frameworks, leaving room for possible software gains.

The RTX 4090 remains the speed leader among 24 GB consumer NVIDIA cards because it has the highest bandwidth in that group. Its mature CUDA software uses more of what the hardware can do than the software for newer cards does today. The 5090’s single-user advantage is meaningful, but not transformative for most solo workloads. The 3090 has the same 24 GB of memory with GDDR6X, so it can fit the same range of models. For buyers comparing these cards, the distinction is therefore speed and price.

The RTX 4070 Super and RTX 4060 Ti serve smaller workloads. Their respective 12 GB and 16 GB capacities support small and mid-sized models at comfortable speeds, but larger models encounter hard out-of-memory limits. The 4070 Super’s roughly 504 GB/s of bandwidth is about half the 4090’s, which fits with its roughly half share of token throughput when the tests use similar conditions. The 4060 Ti has more memory, which gives it more room for aggressive quantization. Partial offload can extend what runs, but its speed penalty can make the result closer to CPU-only performance than GPU performance.

The RX 7900 XTX provides 24 GB at a lower price point than either the 4090 or 3090, with the same model-fit range. Software needs a closer look here than on NVIDIA. The Vulkan path in llama.cpp can outperform ROCm on the same card for pure inference, so AMD users should benchmark both. ROCm trails NVIDIA in training tooling depth, FP8 maturity, and some framework integrations. Those limitations make the intended backend part of the purchase decision for budget inference.

The M3 Max with 64 GB and Strix Halo, or Ryzen AI Max+, with up to 128 GB take the capacity-first route. Their shared memory pools can hold very large models that will not fit entirely on a 24 GB discrete GPU. Smaller-model throughput is the trade-off. Unified memory bandwidth is lower than discrete GDDR6X or GDDR7 bandwidth, so the advantage is access to larger models rather than speed on smaller ones at a comparable price. MLX-LM is the preferred inference stack for Apple Silicon, making framework choice important here too.

The PRO 6000 occupies a different purchasing tier. Its 96 GB makes it the only single tower-friendly discrete PCIe card in this comparison that can hold a very large model at high precision with room for substantial context. That distinction concerns capacity alone. The same bandwidth rules apply to it, even when the workload is too large for consumer cards.

MoE architecture and FP4 shift the bandwidth math

Sparse mixture-of-experts models change how many weights have to be read to generate a token. Unlike the dense-model loop described earlier, an MoE model activates only a fraction of its total parameters for each token. Total size and active work therefore diverge. A sparse model with a moderate active parameter count can generate substantially more tokens per second than a dense model of similar total size, at the same quantization on the same card. Its total size still presents a capacity problem, but it no longer predicts per-token bandwidth demand in the same way.

FP4 changes the bytes-per-weight side of the equation. Available with Blackwell on the RTX 5090 and above, it halves weight storage relative to INT8. TensorRT-LLM and TensorRT results now include the 5090’s FP4 support, while mainstream framework support is still catching up. Getting that benefit takes kernels that work well as well as support for the format. When the budget allows, these developments favor newer architecture and FP4 capability rather than delaying a purchase, because better support can improve bandwidth efficiency without changing the card’s physical memory bus.

Making the GPU selection decision with bandwidth as the organizing principle

Establish the VRAM floor from the target model’s parameter count and intended quantization, then add meaningful space for KV cache and activation buffers. Workloads with a long context need more room than those with a short one. Weights-only figures are a baseline. VRAM calculators that account for real file sizes and hardware checkers that test a specific GPU against a specific model help turn that baseline into a useful filter.

Of the cards that pass, compare the memory bandwidth of those that fit the budget. For batch-size-1 generation, the card with the most bandwidth should be fastest when the software and workload conditions are similar. For multi-user serving, arithmetic throughput belongs back in the comparison because batching can shift the bottleneck. Before choosing, match the benchmark conditions to how the card will be used and test the backend options for that work, including Vulkan and ROCm on AMD. The resulting decision rests on what must fit and what must move for each token, rather than on the largest compute figure printed on the box.