Open-Weight Models That Run on 8GB VRAM

How 8GB gets consumed: weights, KV cache, and runtime overhead

8GB of VRAM is still the most common ceiling in consumer graphics cards. The RTX 3060, the RTX 3070, the RTX 4060, AMD's RX 7600: tens of millions of people run exactly this budget, and none of them are locked out of local AI because of it. But which models actually make good use of that 8GB without falling over the moment a conversation runs long? That list is narrower than most people expect, and getting it wrong is the single biggest reason people give up on local models too early.

Two camps have formed around this. One says 16GB is the real floor now for anything you'd want to use daily. The other points to architecture changes (hybrid attention, sparse designs) that have cut the per-token memory cost enough to keep 8GB genuinely useful. Neither camp has it fully right. The honest answer depends on the specific model, the quantization format, and how much context you're asking it to hold. That's what the rest of this piece works through.

Three things compete for that 8GB, and most people only budget for one of them: the quantized weights themselves. The KV cache grows the longer a conversation runs. Runtime plus framebuffer overhead usually adds 0.5 to 1GB just to keep the software running.

Almost everyone makes a mistake in their first week with local models. They find a model listed at "under 8GB" and treat that number as the whole budget. It never is, and that gap is the mistake itself. Fill the card with weights alone, and the first long prompt starts pushing layers onto the CPU. The fast local model that ran great for five minutes starts crawling, and the person running it usually blames the model instead of the math.

Quantization is the main lever for controlling weight size. Full-precision FP16 or FP32 weights don't come close to fitting on an 8GB card, not even for a modest model. Dropping from FP16 to Q4_K_M cuts the size by roughly 72%, so a 14B model that needed 28GB now needs around 8GB, while retaining most of the original quality. That's why Q4_K_M is the default format for anyone working inside 8GB. A practical rule of thumb: take the Q4 file size, add some headroom for overhead, and that's the real baseline. Pushing the context window past 32K tokens makes that number climb again.

Whether a model works on your card has nothing to do with the spec sheet. Two models can share the same parameter count and the same quantization format and still behave completely differently once a real conversation starts, because the KV cache scales by architecture, not by size. Some designs cost around 15 KiB per token of cache. Others cost closer to 160 KiB per token, over ten times more for the identical conversation length. Whether an 8GB card holds a full 32K-token conversation or chokes after a few exchanges depends on that gap.

Older, conventional 8B models pay for this the hardest. Conventional 8B models pay for this the hardest: an architecture with high per-token cache cost can consume the entire 8GB budget on cache alone at 32K context, before a single byte of model weight even gets counted. The whole card, gone, on cache. No room left for the weights that were supposed to be doing the actual work.

Qwen3.5's linear-plus-full-attention layout and Gemma 4's hybrid attention layout aren't efficiency tweaks bolted on after the fact. Hybrid attention changes what fits inside a fixed memory budget, and that is why they exist. Do the KV cache math before trusting a model to work for you. File size alone tells you almost nothing about whether a long conversation will actually run, and that's the mistake that sends people back to the cloud thinking local models just can't handle real use.

The model shortlist: eight open-weight options verified to run on 8GB, with the tradeoffs each one makes

Qwen3.5-9B is the strongest overall pick for an 8GB card, and it isn't close. Released March 2, 2026 by Alibaba's Qwen team under Apache 2.0, it's a 9B model that quantizes down to 6.6GB at Q4_K_M (ollama pull qwen3.5:9b). It's the only model in this class that stays fully on the GPU at 32K context on an 8GB card, with speed barely shifting between a 4K prompt and a 32K one. If someone is only going to try one model from this list, this is the one.

Architecture explains why. The architecture mixes linear-attention blocks that never need a growing cache with a smaller number of traditional KV-cache layers, and that low per-token cost comes almost entirely from that split. Context runs natively to 262K tokens and stretches to 1M. A "Thinking Preservation" feature keeps reasoning traces across turns, cutting repeated computation in multi-step agent tasks, and the model handles text, image, and video input natively across 201 languages.

On the Artificial Analysis Intelligence Index, Qwen3.5-9B scores 32.4, a 38% lead over the next model on this list, GLM-4.6V, at 23.4. It also posts 82.5% on MMLU-Pro, 81.7% on GPQA Diamond, 82.7% on LiveCodeBench v6, 77.2% on SWE-bench Verified, and 59.3% on Terminal-Bench 2.0. Those scores come from a full-precision version served through an API, so a quantized local build at Q4_K_M won't hit those exact numbers. It'll land close, though.

Gemma 4 12B, in QAT or IQ4_XS form, is the largest dense model that actually fits on 8GB, and "fits" is doing real work in that sentence. Released June 3, 2026 by Google DeepMind under Apache 2.0 (a real change from the more restrictive license on earlier Gemma releases), the family ships in five sizes: E2B, E4B, a 12B Unified model, a 26B-A4B MoE, and a 31B dense variant. At IQ4_XS, the 12B needs about 6.78GB of VRAM at baseline and roughly 7.4GB in practice, handling text, image, and audio across 140-plus languages with a 256K context window. At Q4_K_M it climbs to 7.66GB, which leaves almost no headroom for long context and can force layers off the GPU.

Call the 12B a tight fit, not a comfortable one. Independent benchmarks put MMLU somewhere in the 71 to 75% range, GSM8K math reasoning around 82%, and HumanEval coding at 68.5%. Reach for this one when the dense architecture or Google's training approach specifically matters to the task, and go in knowing long conversations will strain the card. For most daily use, Qwen3.5-9B does the job with less risk of spilling to CPU.

Two smaller Gemma 4 variants solve a different problem than the 12B does. The E4B edge model, at 4B parameters, needs only about 4.0GB of VRAM at Q4, leaving room to run an embedding model alongside it. It's multimodal with 128K context, and it's the right call when the plan involves running more than one model inside the same VRAM budget. The E2B edge model runs 5.1B parameters at 4.73GB in 4-bit, supports reasoning, tool use, and vision, and is noticeably less prone to over-reasoning than some of its peers. It also works as a system-RAM option for anyone without a discrete GPU.

DeepSeek-R1-Distill, in its 7B and 8B forms, is the pick when the job is math or hard multi-step reasoning. Only then. It's distilled from DeepSeek-R1 onto a Qwen2.5-Math-7B base, released under the MIT license. The 7B distill runs about 4GB at Q4, and the 8B variant uses approximately 5.0GB of VRAM at Q4_K_M.

It behaves nothing like a normal chat model. It thinks out loud, generating far more tokens per answer as it works through a problem step by step, and DeepSeek reports strong results on math, logic, and programming benchmarks relative to comparable base models. That verbosity has a real cost, though. If it is asked a quick factual question, or handed a paragraph to rewrite, it will still churn through extended reasoning before answering, burning time the task never needed. This is where a lot of people get the choice backwards. They grab the reasoning model for everyday chat because the benchmarks look better, then wonder why every answer takes three times as long to show up. Pick DeepSeek-R1-Distill over Qwen3.5-9B specifically for competition-level math or structured logic, and leave it out of daily use.

Phi-4-mini takes the opposite bet: smallest footprint, most reasoning squeezed out of each parameter. It's a 3.8B model from Microsoft's Phi-4 family, first released February 2025 under the MIT license, with a 128K context window. As of September 2026, Microsoft hasn't shipped a Phi-5 or a Phi-4.1, so this is still the current small on-device option from that team, even with other labs iterating faster around it.

At Q4_K_M it needs only about 3GB of VRAM, and even at full FP16 precision it stays around 7.6GB, so it runs on almost any modern GPU, including cards under the 8GB mark. It generates over 60 tokens per second at that same roughly 3GB footprint. MMLU is around 67.3% in a 5-shot setup, trailing Llama 3.1 8B on the same benchmark, but Phi-4-mini gets there on about half the memory and twice the speed. Against other 7 to 8B models generally, it stays within 4 to 5 MMLU points while running at roughly half the parameter count. Its strengths are throughput, consistency, and battery life on laptops. It struggles on complex, multi-step reasoning chains, so don't hand it DeepSeek's job.

It's also genuinely portable: smartphones through MLC-LLM or MediaPipe, a Raspberry Pi 5, embedded systems with 4GB or more of RAM, even a web browser through WebLLM. Anyone who wants a reasoning-focused variant can look at Phi-4-mini-reasoning as a direct spin-off, while Phi-4-reasoning is a separate larger model built from the base Phi-4, not the mini.

The stretch tier: MoE models that reach above 8GB by offloading

A MoE model activates only a slice of its total parameters for any given token, so a model with a huge total parameter count can run at speeds closer to what its much smaller active count would suggest, even with part of it sitting off the GPU. That raises the obvious next question: how far can that trick actually get pushed on an 8GB card?

K2 Horizon MoVA 36B-A4B, licensed under Apache 2.0, holds 36B total parameters with only 4B active at a time. It's the highest-scoring model an 8GB machine can realistically run, posting a 26 on the Artificial Analysis Intelligence Index. Getting there means IQ2_XXS quantization, an aggressive compression format even by the standards of this piece, and a dependency on offloading part of the model to system RAM, with 32GB of system RAM required to make that work.

That's the trade in plain terms: more raw capability than anything else on this list, bought with a compression setting cutting closer to the bone than anything else here, and a setup that only works if offloading behaves. Most people chasing this model are chasing a benchmark number instead of a workflow they'll actually stick with. For someone chasing the absolute ceiling of an 8GB card, and willing to tune around the rough edges to get there, this is as far as it goes. For everyone else, skip it. The dense and hybrid models covered above run cleaner, run faster, and demand a fraction of the fiddling this one requires.