Summarization Quality Benchmarks for Local AI Models

Summarization benchmarks for local AI models need to measure more than resemblance to a reference answer. A reliable evaluation separates groundedness, completeness, factual accuracy, and instruction-following, then uses compression as a low-cost monitoring signal.

Why single-metric summarization scoring breaks down for local deployments

Developers now run LLM workloads on their own hardware, so the quality of summaries matters in day-to-day work as well as research. These teams need an evaluation they can trust when a frontier API is not an available fallback. Modern local models paraphrase, restructure, and synthesize, while ROUGE measures overlap with a reference summary. That mismatch lets useful paraphrases look weak and flawed summaries look strong.

The danger is clearest when a summary invents a detail. A confidently invented sentence can share n-grams with the reference and receive credit because ROUGE never checks the source document. Similarly, a summary twice the specified length can benefit from including the right tokens even though ROUGE-L does not test the length instruction. Neither failure is a minor stylistic disagreement. Both mean the model has failed part of the task while the metric can still look reassuring.

BERTScore improves on word overlap by using contextualized embeddings to capture semantic similarity. G-Eval research found that ROUGE-L and BERTScore had very low correlation with human judgments of summary quality. Keep these metrics while limiting their authority over the final judgment.

The four dimensions that together constitute a reliable summarization signal

The replacement is a framework with four distinct dimensions, namely groundedness, completeness, factual accuracy, and instruction-following. Groundedness asks whether every asserted fact is supported by the source. Factual accuracy asks whether the facts in the summary keep the numbers, named entities, and causal relationships correct. These are related checks, but keeping them explicit makes the evaluation more precise than a general impression of correctness.

Completeness asks whether the summary retains the information the source establishes as important. A terse summary can achieve strong compression simply by leaving out too much, so brevity cannot stand in for coverage. Instruction-following asks whether the output has the length and format that were requested. A factually sound summary can still fail this dimension if it ignores the form the reader needs. Compression therefore matters, but its value depends on what survives and whether the result meets the request.

Cisco's Webex Suite AI evaluation paper separates accuracy, covering groundedness and factual correctness, from retention, covering completeness and coverage. Its benchmark found that models differed significantly in retention but not in accuracy, which shows why those properties need to be kept separate. Min et al.'s 2025 ACL research, “Towards multi-dimensional evaluation of LLM summarization across domains and languages”, formalized the broader move beyond ROUGE. The practical implication is to preserve the separate signals even when a composite score is useful for comparison.

Standard datasets and where each one tests evaluation pressure

A benchmark corpus is useful only in relation to the property it tests. CNN/DailyMail with ROUGE-L remains a canonical reference point for local deployment research, serving as a compatibility baseline and a way to detect regressions. XSum adds pressure on abstractive summarization through its single-sentence format, which requires genuine paraphrase.

A suite spanning 2,400 documents across eight domains includes news, legal, medical, scientific, financial, conversational transcripts, code documentation, and customer support tickets. Its composite combines ROUGE-L, BERTScore, and a custom fact-preservation metric. Meeting corpora including ICSI, AMI, QMSum, MeetingBank, and ExplainMeetSum place particular pressure on completeness and retention. These are different tests from condensing a conventional document.

Specialized resources put faithfulness under closer scrutiny. Vectara's hallucination leaderboard has tracked summarization hallucination rates since 2023, offering an updated signal. A 2025 framework published in npj Digital Medicine addresses clinical safety and hallucinations in medical text summarisation. FaithBench, presented at NAACL 2025, provides human hallucination annotations for summaries from ten modern LLMs across eight model families. Those labels give a faithfulness judge ground truth to check whether it catches the failures people find, something reference-comparison scores do not offer.

How to evaluate groundedness and faithfulness under local-hardware constraints

Human annotation at scale is expensive, while cheaper methods can be noisy. LLM-as-judge offers a practical substitute for repetitive scoring in support of human judgment. The human role is to set up the claim decomposition and interpret the consistency signals. The model's role is to perform the repeated per-claim checks.

Cisco's Webex Suite AI system makes this division concrete through claim-grounded scoring. Ground-truth artifacts are decomposed into discrete claims, and the judge evaluates each claim against the source. This isolates factual failures that a holistic score can hide. A custom fact-preservation metric serves a related purpose by making faithfulness an explicit part of evaluation rather than an assumed benefit of semantic similarity. Its design can reflect the failures that have reached production in the relevant domain.

Research on LLM judge reliability found that a large open-weight model correlated more strongly with GPT-4o's judgments than with human judgments. FaithJudge with o3-mini-high also tends to underpredict hallucinations relative to FaithBench's human labels, including for Command-R, Mistral, and Qwen. Multiple consistency runs are therefore a practical mitigation, while single-run reporting without consistency measures can mislead. Repetition helps expose unstable judgments, but does not make human or domain-expert review unnecessary.

What the benchmark numbers actually show across open-weight model families

Llama 3.3 Q4_K_M recorded ROUGE-L of 0.418, factual recall of 89.1%, and a hallucination rate of 1.4% in the 2026 benchmark results. Qwen3 Q4_K_M had a ROUGE-L score of 0.412, factual recall of 87.3%, and a hallucination rate of 2.1%. The reported ROUGE-L quality gap was under 3%, while Llama 3.3 70B required more than double the VRAM of Qwen3 32B. The decision therefore depends on the cost of the score margin.

Other results bring different constraints into view. Gemma 3 Q5_K_M had near-flagship coherence at 38 tokens/sec on an RTX 4070 Ti, showing what it could do on mid-range hardware. With map-reduce chunking, Qwen3 Q4_K_M had higher factual recall than every other small model in the long-document benchmark. Meanwhile, Qwen 2.5 and Gemma 3 stand out for following explicit summarization format and length instructions.

Qwen3-30B-A3B-Instruct-2507 adds an architectural distinction to the comparison. It is a model that activates only a subset of its parameters, with 30.5 billion total parameters and only 3.3 billion activated parameters, with significant improvements in text comprehension, logical reasoning, and instruction following for summarization. That makes the amount of active computation matter, while teams still need to check what their local hardware can handle. Across the 2,400-document comparison, the composite spread between the cheapest and most expensive models was small. Taken together, these results make faithfulness, instruction-following, and deployment configuration more useful decision points than raw ROUGE alone.

How quantization, context length, and temperature interact with summarization quality

Once a model is selected, configuration becomes part of the quality evaluation. The main levers are quantization level, effective context window, and inference temperature. Architectures do not all lose the same amount of quality at Q4. Testing Q4_K_M against Q8_0 shows whether the quality gap is meaningful or negligible for the chosen model and workload.

Context capacity needs the same practical test. A nominal context window can overstate effective summarization capacity, with attention degradation and measurable recall loss on very long multi-document inputs. Map-reduce chunking addresses this by splitting documents into segments, summarizing each segment, and then summarizing those summaries. The Qwen3 factual-recall result illustrates why this configuration choice can matter as much as the model ranking.

Temperature has its most direct effect on groundedness and focus. For summarization, the practical range is 0.1–0.3. Higher temperatures increase the probability of hallucinated details and unfocused output. Quantization, chunking, and temperature should all be tested on the same separate dimensions used to compare models.

Using compression ratio as a production monitoring signal

Offline evaluation establishes a baseline, but deployment changes can shift behavior after that baseline is set. Model updates, quantization changes, and hardware swaps can all alter summarization output. Compression ratio, which is output tokens divided by input tokens, is a cheap way to spot a drop in quality without running a judge on every request. Because it is model-agnostic, it can remain useful as the underlying deployment changes. It measures the shape of the output, leaving its truth to be assessed separately.

That distinction defines its proper role. Compression ratio is a tripwire rather than a diagnosis, because a change does not identify which quality dimension failed. A flagged document needs to be checked on all four dimensions to tell whether it has missing coverage, poor instruction-following, or factual problems. Conversely, a plausible ratio cannot certify a faithful summary. The value lies in directing more expensive attention.

Cisco's Webex Suite AI architecture provides a broader model for this monitoring loop. It persists transcripts, ground-truth artifacts, candidate summaries, and judge outputs, while its online monitoring layer surfaces directional movement and hard regimes without exposing customer data. That online evidence is not itself a benchmark. Instead, it connects production behavior to offline evaluation, so monitoring alerts can lead to reusable tests.

Building the practical evaluation stack: what to measure, in what order

Keep ROUGE-L and BERTScore on a held-out reference set as regression detectors between model versions. Add compression ratio monitoring to production traffic as the live tripwire. This first layer establishes whether familiar behavior has shifted. It does not decide whether a summary is safe or complete.

Next, run offline evaluation at regular intervals on a corpus from the domain being tested. General pipelines need news and conversational material, while legal, medical, or financial use calls for the relevant subsets. Use claim-grounded judging to score groundedness and factual accuracy, with multiple runs to check consistency. Score completeness and instruction-following separately, and add a custom fact-preservation metric calibrated to the domain's observed failures. This layer explains changes that the cheaper signals can only flag.

Finally, have people or domain experts audit the difficult cases. FaithBench's human labels show why judge outputs need that anchor, particularly where hallucinations are under-flagged. Cisco's benchmark identified Whitehouse press briefings as an accuracy-hard regime, illustrating why difficult domain-specific conditions should be mapped during initial evaluation. Incidents found in production can then become new regression cases in the offline corpus.