First Token Latency vs Throughput Trade-Offs in Local Inference
Optimizing for speed requires choosing between fast first tokens or high output volume.
Tobias Gerhart
Section
Model Benchmarks
7 stories in Model Benchmarks.
Tokens per Second Benchmarks Across Consumer GPUs
Memory bandwidth, not raw compute power, determines how fast GPUs generate tokens.
Tobias Gerhart
Reasoning Model Benchmarks for Local Deployment
Benchmarking reasoning models requires measuring hardware constraints, not just accuracy scores.
Solange Okafor
Summarization Quality Benchmarks for Local AI Models
Separating groundedness, completeness, and factual accuracy reveals what ROUGE scores miss.
Tobias Gerhart
Coding Task Performance of Small Local Models
Small local models excel at routine coding tasks but struggle with complex multi-step work.
Marcus Llewellyn
Instruction-Following Accuracy of Local Models on Structured Tasks
Local models fail structured tasks through logic, timing, and uncertainty—not just format.
Priya Venkataraman
Context Length Limits That Matter for Real Workflows
Effective context capacity runs 60-70 percent of advertised maximums in production systems.
Solange Okafor