First Token Latency vs Throughput Trade-Offs in Local Inference

Local inference has two different meanings of fast, namely how soon an answer starts and how much output the system can produce. First-token latency and throughput pull against each other because prompt processing and token generation compete for shared hardware in different ways. The right optimization therefore starts with what the application is waiting for.

Why prefill and decode are structurally different computations

Transformer inference begins with prefill, which processes the entire input prompt in parallel and builds the key-value, or KV, cache before emitting an output token. Most of its work is matrix multiplications across the input sequence, which makes it compute-bound. Decode then generates output tokens one at a time, with each token depending on what came before it. The bottleneck changes at that boundary. Decode must read model weights and the KV cache for every token, so memory bandwidth constrains generation.

This explains why a system can start quickly but generate slowly, or take longer to start and then keep output high. The same hardware must support two different resource demands. Lyceum's inference benchmark separates time to first token, or TTFT, from inter-token latency and tokens-per-second for this reason. TTFT includes network round-trip time, queue wait, and prompt prefill, while inter-token latency is the time between generated tokens. Aggregate tokens-per-second describes the system's output capacity.

How prompt length and batch size each amplify the tension in a different direction

A longer prompt increases the work that must finish before generation can begin. Full attention has quadratic complexity in sequence length, written as O(n²), so prefill takes much more work as inputs grow. The prompt is not always freely adjustable. A RAG pipeline that injects retrieved documents has a different input requirement from a short conversational probe. The Lyceum benchmark says its short probe is close to each model's TTFT floor, while requests with many documents will be well above it.

Batch size adds a different pressure: it can raise aggregate throughput but make each request wait longer. Under static batching, requests wait for a full batch to assemble before prefill starts, and that queue time adds directly to TTFT. Once decoding begins, the sequences share memory bandwidth at every step. Inter-token latency can therefore degrade before TTFT looks troubling.

This creates an important asymmetry between the two levers. Prompt length is often an application requirement, while batch size is an engineering choice about how to serve that requirement. The University of Michigan's GPU-to-Grid research treats batch size as the primary GPU-side control knob because it governs power consumption, inference latency, and token throughput together. Changing it changes how the system uses hardware and how individual requests experience that use.

Why the same hardware and model produce very different experiences depending on what the application is waiting for

What does the application count as progress? For a streaming chatbot, the first visible token ends the initial wait, making TTFT central to the experience. Once generation is faster than a human reading rate of roughly 4–5 tokens per second, the reader will barely notice any further speed. The Lyceum benchmark accordingly treats TTFT as the application-facing latency number and tokens-per-second as an internal cost and capacity number. A faster stream does little to compensate for a blank screen at the start.

A pipeline waiting for a complete answer has the opposite priority. A RAG retrieval-and-synthesis step or batch enrichment job cannot advance merely because the first token has appeared. Consider a model with 0.4s TTFT and a generation rate of 50 tokens/second on a 500-token response. A model with 0.8s TTFT but a much higher generation rate can finish much sooner.

Sequential agents complicate this contrast because every call adds another first-token wait. Even with high generation throughput, the wait across a multi-step pipeline adds up. A pipeline can also take a long time to finish if its internal responses are long, even when the first token arrives quickly. Both costs can accumulate within one application. Which metric deserves priority depends on where the application can use partial output and where it must wait for completion.

A workload taxonomy that maps application type to the metric that actually matters

TTFT-dominant workloads include inline code completion, streaming chat, voice agents, and real-time trading signals. They depend on a prompt start. These workloads are typically low-concurrency and short-prompt. That combination is where local inference has its clearest latency advantage over cloud APIs, because the workload places relatively little pressure on prefill and shared serving capacity.

Throughput-dominant workloads include batch enrichment, overnight evaluation runs, RAG retrieval-and-synthesis, and internal tools at low-to-moderate query rates. These can tolerate seconds of TTFT if the system processes the queue economically. Cost-per-token then measures how well the system performs. The distinction is not whether a person ultimately uses the result, but whether early output has value before the full response is ready.

Mixed workloads include multi-step agentic pipelines and document processing with streaming previews. Their user-facing hops can be TTFT-dominant while internal hops depend on throughput, so one label need not govern the whole application. Load also changes the practical result. Local inference's advantage over cloud APIs is large at low utilization, but tail TTFT can widen significantly as GPU utilization rises under continuous batching. A workload category identifies the priority without guaranteeing that the deployment will meet it.

What the serving stack does to TTFT and throughput before any tuning starts

The inference runtime already shapes both metrics through its handling of memory and request scheduling. PagedAttention, the approach underlying vLLM, manages the KV cache in fixed-size blocks rather than reserving contiguous regions for each request. This is paging applied to inference memory. Tighter packing supports larger batches without exhausting memory, and the Lyceum benchmark reports substantially higher throughput than earlier serving systems at the same latency level.

Continuous batching, available in engines such as vLLM and SGLang, allows new requests to join mid-batch rather than waiting for a static batch to form or finish. That reduces a source of queue delay under load. It changes when work can begin while bandwidth competition during decode remains.

Apple Silicon offers a concrete example of that distinction. Across a range of model sizes, vllm-mlx achieves substantially higher throughput than llama.cpp by adding continuous batching that llama.cpp lacks for concurrent requests. Yet llama.cpp records the fastest TTFT of any tested backend on the Qwen3.5 family, an advantage it cannot sustain under concurrency. Neither result cancels out the other. Opper's 2026 router latency benchmark likewise finds that changing the backend leads to a larger throughput spread than changing the model, with no overlap between backend confidence intervals, so serving speed depends on more than model choice.

Techniques that shift the prefill-decode boundary, and what each one actually trades away

Chunked prefill changes how long prompt processing can occupy the GPU without interruption. The Sarathi-Serve paper at OSDI 2024 splits large prompts into 256–512 token chunks and interleaves them with decode iterations. This eliminates decode stalls caused by long prefills monopolizing the device. The price is slightly higher average TTFT. In exchange, the system gains dramatically lower tail latency and higher throughput without additional hardware, making this a good-enough intervention for most workloads.

Disaggregation takes a different approach by routing prefill and decode to separate processes or hardware. Rather than sharing GPU time more carefully, it separates the phases so TTFT and inter-token latency can be tuned independently. vLLM's single-node disaggregation using MORI-IO in April 2026 illustrates the benefit. Standard aggregated serving failed inter-token latency service-level objectives, while both disaggregated modes eliminated those violations entirely.

The distinction is explicit in vLLM's documentation, which states that disaggregation does not improve throughput. Its costs include RDMA infrastructure requirements and KV-transfer latency on the critical path. Separation is not a free improvement either. Chunked prefill adds a little to the average wait for the first token to keep decode moving on existing hardware, while disaggregation adds transfer and operational overhead so each phase can be controlled on its own. The choice is real when strict tail inter-token latency requirements demand more than interleaving can provide.

Quantization moves a different constraint by reducing memory footprint, allowing more sequences to fit in the KV cache simultaneously and improving throughput. Aggressive quantization can, however, degrade quality and raise retry rates, partly offsetting the cost gain. The hardware-specific options in this trade are FP8 on Hopper and Blackwell, INT4 with AWQ for weight-only quantization, and INT8 with SmoothQuant for older-generation GPUs. Those choices ease memory pressure, but prefill and decode still need to be balanced.

How hardware architecture is beginning to dissolve the trade-off at the infrastructure level

Hardware vendors are starting to design different hardware for the two phases, going beyond changes to how they schedule the work. At GTC 2026, NVIDIA announced a direction toward disaggregated inference with prefill on compute-optimized GPUs and decode FFN/attention phases on Groq LPX static-memory-based LPUs. The physical split follows the computational one.

Consumer hardware also changes who can run local inference with useful throughput. NVIDIA's Blackwell consumer GPUs, the RTX 50-series, include native NVFP4 inference support. The RTX 5090 offers up to 1.8 TB/s memory bandwidth at approximately $2,000, versus a multiple of that for data-center-class hardware. Access to substantial bandwidth is becoming less restricted. That does not establish a universal latency or throughput result, because the runtime, request mix, and concurrency still determine how that bandwidth is used.

These changes could ease the trade-off, though it would still remain. The application still has to set its own priorities. Understanding the prefill-decode distinction remains necessary to judge whether a new architecture addresses the workload's actual bottleneck.

A decision framework for choosing which metric to optimize on a given local inference workload

Tuning parallelism on the same hardware, model, and workload cut TTFT from 995ms to 304ms at 23.6 requests per second. Smarter EPP request routing cut it further to 287ms. New hardware was not the first requirement. Identify the workload's priority and serving constraints before reaching for a different architecture.

The initial metric choice depends on whether the user or downstream system sees the first token before the response completes. If yes, TTFT is the primary metric, so the priorities are faster prefill and minimal queuing. If no, throughput is primary, so aggressive batching and higher TTFT are acceptable. For mixed pipelines, that choice has to be made at each hop. A streaming preview and the internal processing behind it need not receive the same treatment.

The second question concerns expected concurrency, because it determines which serving approach can support that priority. At low concurrency, latency-optimized runtimes with continuous batching disabled or lightly configured can serve TTFT well, including llama.cpp-class tools. Higher concurrency calls for continuous batching and PagedAttention to prevent TTFT degradation under load. Batch configuration belongs in that same decision. It sets the balance among shared capacity, request latency, and GPU power.

Short, consistent prompts suit standard prefill, with TTFT near its hardware floor. Long or variable prompts make chunked prefill the lowest-cost intervention. Full disaggregation is justified only when tail inter-token latency objectives are strict and the infrastructure can absorb RDMA requirements.

AI-native tooling can apply this framework automatically when it understands request context and workload patterns. It can reason about the dominant metric for a given call rather than requiring manual reconfiguration for every workload type. That automation still needs an engineering objective. Its useful role is to apply the chosen priorities consistently, preserving the judgment about what the application is waiting for.