Apple Silicon for Local AI Model Inference

Local AI inference lives or dies on memory, not raw compute power. Apple Silicon's unified memory architecture changes the math entirely, and understanding why requires looking past core counts toward gigabytes and gigabytes-per-second.

Memory capacity, not compute cores, as the binding constraint for local LLM inference

Running a large language model on a local machine is, at its core, a memory problem. For every single token the model produces, its entire set of weights has to move from memory into the compute units doing the math. That's not a one-time cost paid at startup. It happens over and over, once per token, for the whole length of a response.

So what happens when the weights don't fit in memory? The model simply doesn't load. There's no partial credit, no degraded mode where it runs a little slower. It fails outright. That's the first hurdle, and it's a hard one: capacity before speed, always.

On a discrete GPU system, that ceiling is set by VRAM. Only what's on the graphics card counts.

Once a model does fit, a second question takes over: how fast can it actually generate text? That's the real-world yardstick. A chip producing 12 tokens per second isn't a disappointment, it's already outpacing a human reader Apple Silicon 2026: M6 to M5 Ultra for Local LLMs.

So there are two distinct failure modes a practitioner runs into. One: the model won't load, a capacity problem. Both matter, and this piece works through them in that order, because capacity is the gate you have to pass before speed becomes relevant at all. On a discrete GPU system, the memory ceiling is the VRAM (the system's DDR5 RAM is irrelevant to the GPU, no matter how large). Human reading speed is roughly 4–7 tokens per second, established early as the practical benchmark against which all tok/s figures should be judged.

Unified memory and the VRAM ceiling that constrains discrete GPU systems

Picture a typical workstation built around a discrete NVIDIA GPU, where the hard ceiling on a top consumer NVIDIA GPU is 24 GB VRAM. The CPU has its own pool of RAM. The GPU has its own, separate pool of VRAM, connected to the rest of the system over PCIe. The GPU can only ever read and write inside its own pool. Even the flagship consumer card tops out at 24 GB of VRAM, and that number is fixed the day the card is manufactured.

Apple's architecture works differently. The CPU, GPU, and Neural Engine all draw from one single physical pool of memory, at full bandwidth, with nothing being copied or shuffled between separate zones. There's no ceiling below the total amount of RAM installed in the machine.

That has a very concrete consequence. A Mac with 64 GB of unified memory can load a 70-billion-parameter model, quantized down to 4 bits, and start producing tokens right away. That same model has nowhere to go on a consumer NVIDIA card. It won't fit, full stop, no matter how the card is configured.

Part of why this works so cleanly is that unified memory allows memory-mapped loading. The model doesn't need to be shuttled from one pool into another before inference can begin; it's already sitting where the GPU can reach it. And this isn't some new feature bolted onto the latest chip. It's a structural property that's been true across the entire M-series lineup going back to the original M1.

Independent researchers have noticed the same thing. That framing did not come from Apple's marketing department; it came from people who build inference software for a living.

Which sets up the rule that carries through the rest of this piece: when sizing up any Apple Silicon chip for inference work, unified memory capacity and bandwidth are the specs that matter first. GPU core count comes second. The BaseRT paper (arxiv) states that Apple's unified memory architecture "removed the traditional CPU–GPU memory segregation, giving the GPU direct high-bandwidth access," cited here as independent academic framing, not Apple marketing.

Memory bandwidth as the second axis: why GB/s determines tokens per second once a model fits

Suppose the model fits. What decides how fast it runs?

Almost entirely, it comes down to how quickly the weights can be pulled out of memory. Not the number of GPU cores sitting idle, waiting for data. The bandwidth of the memory bus itself.

A worked example makes the arithmetic concrete. To generate 100 tokens per second, the system has to sustain 550 GB/s of memory reads, since every token requires reading through the full set of weights again Apple Silicon for AI: M1 to M6 Compared (2026) - llmcheck Hey Eduardo. That's nearly saturating the M5 Max's roughly 600 GB/s ceiling, on a mid-sized model Apple Silicon for AI: M1 to M6 Compared (2026) - llmcheck Hey Eduardo.

This is why throwing more GPU cores at the problem doesn't scale inference speed the way it might scale, say, video rendering.

Across the available data, memory bandwidth correlates with inference speed more strongly than GPU core count, Neural Engine TOPS, or even chip generation. And there's a telling detail buried in how Apple talks about its own hardware: at the announcements for the M6 and M5 Ultra, Apple didn't publish Neural Engine TOPS figures at all. Compare that to Qualcomm and AMD, both of whom lead their AI PC marketing with TOPS numbers front and center. Apple's silence on that spec is itself a signal about where it believes the real bottleneck lives.

Keep the bandwidth figures in mind going forward. They aren't decorative marketing specs. They're the direct ceiling on tokens per second at any given model size, and they're the numbers that actually predict what a chip can do.

The current Apple Silicon lineup mapped to inference use cases, from Mac mini M6 to Mac Studio M5 Ultra

Apple replaced its entire desktop lineup on August 25, 2026, with hardware shipping September 22, 2026, including the Mac mini M6 and M5 Pro and the Mac Studio M5 Max and M5 Ultra Hey Eduardo. The previous generation, the M4 Mac mini, M4 Max Mac Studio, and M3 Ultra Mac Studio, is no longer sold new Hey Eduardo. The top-tier 512 GB M5 Ultra configuration ships a bit later, in late October 2026, while the 256 GB version ships alongside everything else in September Apple Silicon 2026: M6 to M5 Ultra for Local LLMs.

One caveat has to sit over this entire section: none of these new chips have independent, third-party benchmarks yet. Every performance figure attached to the M6, M5 Max, and M5 Ultra at this point is either Apple's own claim or a community projection based on architecture Hey Eduardo. Treat them as provisional until real hardware is in real hands.

With that said, here's how the lineup breaks down. The Mac mini M6, starting at $899, offers up to 32 GB of unified memory at 170 GB/s, with a 12-core GPU and a dual 16-core Neural Engine Apple Silicon 2026: M6 to M5 Ultra for Local LLMs. Its sweet spot is 7B to 14B parameter models https://localaimaster.com/blog/apple-silicon-ai-buying-guide.

A Mac with 64 GB unified memory can load a 70B-parameter model at 4-bit quantization and begin serving tokens immediately, a model that will not fit on any single consumer NVIDIA GPU. Apple hasn't published that bandwidth figure itself; it comes from independent estimates, not an official spec sheet Best Mac for Local AI (2026): Apple Silicon Buying Guide.

That memory ceiling matters because it opens the door to models nothing else on the consumer market can hold. The M5 Ultra is the only single consumer machine that can fit something like DeepSeek-R1 671B, or DeepSeek V4-Flash at native precision, entirely inside unified memory.

RAM is permanent at the point of purchase and cannot be upgraded, with a practical minimum of 24 GB, 36 to 48 GB recommended for developers and agent workloads, and 64 GB or more for future-proofing through 2027-2028. Buy for where the models are going. The M5 Pro (Mac mini, from $1,699 MSRP) offers up to 64 GB unified memory and 307 GB/s bandwidth, with an LLM sweet spot of up to 34B models comfortably and 70B at Q4 with the 64 GB configuration Apple M5. The M5 Max (Mac Studio, from $2,499 MSRP) offers up to 128 GB unified memory and 460 GB/s bandwidth in the 32-core GPU configuration or 614 GB/s in the 40-core GPU configuration, with an LLM sweet spot of 70B models Apple M4 Specs, Benchmarks, Release Dates & Prices Apple M5 Apple Silicon 2026: M6 to M5 Ultra for Local LLMs. The M4 Pro, for reference, offers up to 64 GB, 273 GB/s bandwidth, and a 20-core GPU.

Real inference speeds across the M4 and M5 families

Roughly 10 tokens per second feels responsive Apple M4 Specs, Benchmarks, Release Dates & Prices. Above 30, it feels instant.

The M4 Pro delivers roughly 40 tokens per second on 7B models, 20 on 13B, and 12 on 30B. The M4 Max, with its larger memory pool, manages about 48 on 7B, 26 on 13B, and 10 on 70B models, with tokens per second dropping as model size grows, roughly in line with how much more data has to move through memory for each token. That's the bandwidth framework playing out in real numbers.

The M5 family's figures are community projections, unverified by independent testing at this point.

These are Apple's pre-production figures (claimed to show LLM prompt processing in LM Studio up to 13.5x faster than M1 Mac mini and 4.8x faster than M4 Mac mini), not independently verified. M4 community benchmarks (2026) are presented below. On the M4 (16–32 GB), performance reaches approximately 33 tok/s on 7B models and approximately 17 tok/s on 13B models. On the M5 Pro (64 GB), performance reaches 45–55 tok/s on 8B models and 15–20 tok/s on 34B models. On the M5 Max (128 GB), performance reaches 12–18 tok/s on 70B models On-Device AI Inference 2026: Apple Silicon, NVIDIA & AMD Guide. As a practical illustration, a MacBook Pro M3 with 18 GB unified memory achieves 18–22 tok/s on Qwen3 14B, perfectly usable for development and Q&A work On-Device AI Inference 2026: Apple Silicon, NVIDIA & AMD Guide. Time to first token on local Apple Silicon typically arrives in 15–80 milliseconds, faster than a round-trip to a cloud API, which matters for interactive and agentic workflows.

Where Apple Silicon lags NVIDIA

None of this means Apple Silicon wins every comparison. For an identical model size, NVIDIA's discrete GPUs are faster per token, and by a meaningful margin.

The reason lines up exactly with the bandwidth framework laid out earlier. Apple's memory bandwidth is high by the standards of unified architecture, but it's still lower than the HBM-based bandwidth on NVIDIA's top-end cards, and at matched model sizes, the faster bus simply wins on raw speed. There's no contradiction here, just two different design philosophies optimized for different constraints.

Where does NVIDIA hold a decisive edge? Production inference serving many concurrent users at once, batch processing jobs, and real-time voice AI where latency at scale is the whole game.

But flip the framing to model capacity, and the picture reverses. A Mac mini can run a 33B model that has no path onto any consumer NVIDIA card, because the RTX 4090's 24 GB of VRAM is a wall that can't be moved. Apple wins on capacity per dollar, not on raw speed per token.

The capacity advantage tends to matter more day to day than the raw speed gap. Metal works without any driver setup, no CUDA version conflicts, no chasing down NVIDIA driver updates, which counts for a lot on a team without dedicated infrastructure support. Sources note that annual electricity for 24/7 LLM inference is roughly $35–55 on the Mac mini M6 versus $300–400 on a desktop RTX 4090, a roughly 10x difference in operating expense that compounds over time. For individual developers, researchers, and AI-augmented knowledge workers running one conversation at a time, the 4–7 tok/s threshold means even slower Apple Silicon speeds are practically sufficient, and the capacity advantage affects day-to-day usability more than the speed gap does.

The inference runtime layer: how MLX became the dominant execution path on Apple Silicon

Hardware is only half the story. Software decides how much of that hardware actually gets used.

Apple Silicon offers three distinct paths for running inference: the Neural Engine through Core ML, the GPU through Metal, or the CPU. For general LLM work in 2025 and 2026, GPU via Metal is the primary, recommended route. The Neural Engine is built for CoreML-converted models with fixed computational graphs, and its inflexibility with dynamic shapes and large parameter counts limits how useful it is for the kind of general-purpose LLM inference most people actually want.

At WWDC 2025, Apple made its preference explicit, dedicating three separate sessions to establishing MLX as the framework of choice for LLM inference. Apple also released a Swift API aimed at app developers building on-device features, part of a framework for foundation models introduced alongside it. The split is deliberate: MLX for researchers and advanced developers who want low-level control, Foundation Models for app developers who want something closer to plug-and-play.

MLX earns its position through three specific design choices. It performs true zero-copy operations that take full advantage of unified memory. It uses lazy evaluation, which fuses operations together and cuts down on memory allocation overhead.

That shift is visible in Ollama itself. Version 0.19, released March 30, 2026, switched Ollama's Apple Silicon backend from llama.cpp's Metal implementation over to MLX. Decode speed on identical hardware jumped from 58 tokens per second to 112, a 93% improvement with no change to the hardware, the model, or the quantization level. To be precise, Ollama didn't drop llama.cpp entirely, it kept it for other hardware platforms while using MLX specifically on Apple Silicon, a dual-stack setup rather than a full migration.

The size of that gap depends heavily on model architecture. A concrete case makes this vivid: running the MoE model Qwen3-Coder-30B-A3B on a Mac mini M4 Pro with 64 GB, MLX hit roughly 130 tokens per second against roughly 43 for Ollama's older llama.cpp backend. That's not a marginal win, that's three times the throughput on identical silicon.

MLX isn't strictly better in every scenario, though. Past roughly 30,000 tokens of context, llama.cpp's FlashAttention implementation can actually pull ahead of MLX. That matters for anyone doing long-context research or working through large documents, where context length routinely blows past that threshold. The MLX vs. llama.cpp performance gap is roughly 1.4x to 1.8x on dense models and up to roughly 3x on mixture-of-experts models, because MoE inference is more memory-bound and MLX moves weights more efficiently.

BaseRT and the frontier of native Metal inference: how far above MLX the ceiling extends

If MLX represents the current mainstream ceiling, BaseRT is a look at what sits above it BaseRT: Best-in-Class LLM Inferenceon Apple Silicon via Native Metal. Built by Base Compute in Melbourne and described in a 2026 arxiv paper, BaseRT is a native Metal inference runtime built directly against Metal, rather than wrapping it inside a higher-level framework abstraction.

The measured results back that up.

Why would the MoE prefill gains outpace the decode gains Apple M4 Specs, Benchmarks, Release Dates & Prices Best Mac for Local AI (2026): Apple Silicon Buying Guide? It tracks with the same bandwidth logic running through this entire piece: MoE models only activate a sparse subset of their total weights for any given token, so a runtime that manages memory access more precisely has more room to pull ahead, and that gap is largest exactly where memory movement is most complex. BaseRT is a research project rather than a shipping consumer product, but it marks out how much headroom still exists above what MLX and llama.cpp deliver today, and it's a reasonable bet that some of these techniques find their way into mainstream runtimes before long. Measured results on M3 and M4 Pro devices used Q4 and Q8 quantization across the Qwen3, Llama 3.2, and Gemma 4 model families. Prefill throughput on mixture-of-experts models showed up to a 1.78x improvement over MLX at longer prompt lengths BaseRT: Best-in-Class LLM Inferenceon Apple Silicon via Native Metal.