Gemma 2 for Private Document Summarization

Gemma 2 is Google DeepMind's family of open-weight language models, built in three sizes, 2B, 9B, and 27B parameters, and all three run entirely on local hardware. That last part is the whole story for anyone summarizing sensitive documents: contracts, patient records, deal memos. Google released the 9B and 27B versions on June 27, 2024, then added the 2B variant a month later on July 31, 2024. Gemma 2 draws from the same research lineage as the Gemini models, scaled down into weights anyone can download and run without a server call to Google's infrastructure.

Newer generations exist now. Gemma 3 landed on March 12, 2025, and Gemma 4 followed on April 2, 2026. Neither replaces Gemma 2 for the purposes of this piece. Gemma 2 has over 150 million downloads and something like 70,000 community variants on Hugging Face, which means the tooling, the fine-tuning recipes, and the troubleshooting knowledge are already mature. Gemma 4 only picked up an Apache 2.0 license on April 2, 2026; Gemma 2 still runs under the original Gemma Terms of Use, a detail with real procurement consequences covered below. For a document summarization pipeline that needs to work today, on hardware already sitting in an office, Gemma 2 is the practical starting point.

Why local deployment is the right foundation for summarizing sensitive documents

Sending a document to a cloud API means the text leaves the building. That's true even when the vendor promises not to retain it, not to train on it, not to log it. Every one of those promises is a contractual assurance, not a physical fact, and contracts get breached, misconfigured, or reinterpreted by a new legal team two years later.

Local deployment removes that variable. Data processed on-device, or inside a private cloud perimeter you control, never crosses a third-party boundary, no matter what the document contains. That's a fact about where the bytes physically travel, not a policy. It's a fact about where the bytes physically travel.

Compare that to a hosted model endpoint. Hosting reduces the local infrastructure burden considerably, no GPU procurement, no driver headaches, someone else manages uptime. But it shifts the risk onto network controls, contractual terms, and the vendor's own security posture. That's a real trade-off, not a hidden one, and it matters most in a few fields:

Legal: contracts, discovery documents, privileged communications between counsel and client

For anyone building a retrieval-augmented generation (RAG) pipeline on top of a summarization workflow, research on LLM behavior has found models can reproduce retrieved text verbatim at significant rates. In a RAG setup summarizing internal documents, that means chunks of the source material can show up nearly word-for-word in the output. Local deployment doesn't eliminate that behavior, but it does mean the verbatim leakage stays inside the perimeter instead of traveling to a third-party server alongside the query.

License terms matter here too, and they're easy to skip past. Gemma 2, like Gemma 3, ships under the custom Gemma Terms of Use rather than Apache 2.0 (Apache 2.0 only arrived with Gemma 4). Organizations with strict procurement rules around irrevocable, standard open-source licenses need to read those terms closely before putting Gemma 2 into production, particularly in regulated industries where legal review of every dependency is standard practice.

On the training side, Google filtered Gemma 2's pretraining data to strip out personally identifiable information. Google filtering the pretraining data to strip out personally identifiable information is a reasonable hygiene step. But it doesn't substitute for local deployment. A model trained on clean data can still leak whatever sensitive document you feed it at inference time if that inference happens on someone else's server.

How Gemma 2's architecture shapes its summarization behavior

Gemma 2's 2B and 9B models weren't trained the ordinary way. Instead of learning purely from next-token prediction, they were trained through knowledge distillation, with a larger teacher model shaping their outputs during training. The practical effect: these smaller models perform closer to models two to three times their parameter count than you'd expect from their size alone. That matters enormously for local deployment, where every extra billion parameters costs real VRAM.

The attention mechanism is where things get interesting for document processing specifically. Gemma 2 alternates between two types of attention layer by layer: local sliding window attention, covering a 4,096-token window, and global attention, covering the full 8,192-token context. Every two layers, the model gets a complete view of the document, both the immediate neighborhood around each token and the document-wide structure. That's efficient within the 8K window. It does not, however, stretch that window further. No amount of clever attention scheduling turns 8,192 tokens into roughly double that without retraining the model.

On the memory side, the 9B and 27B models use Grouped-Query Attention (GQA), which cuts the memory bandwidth needed during inference. That's a direct lever on how much VRAM a deployment needs and how fast tokens come out the other end. The 2B model uses Grouped-Query Attention (GQA), the same memory-efficient attention design shared across the family, built for laptop-class hardware where every gigabyte of VRAM counts.

Two more architectural choices affect summarization quality specifically. Logit soft-capping keeps the model's output logits from growing unbounded during training, which improves training stability and contributes to steadier outputs. And the post-training process included work aimed at improving output quality for grounded tasks, behaviors that matter enormously when the task is summarizing a document rather than freely generating text. A model that hedges when uncertain is a model less likely to invent a contract clause that isn't there.

The 9B model trained on 8 trillion tokens, a corpus 30% larger than Gemma 1.1's. That breadth of exposure is a big part of why 9B tends to be the most practical general-purpose choice for summarization: enough pattern coverage to handle varied document types without needing the full 27B footprint.

The 8,192-token ceiling's impact on real documents

All three Gemma 2 sizes share an 8,192-token context window. That's fixed. It doesn't flex at inference time, and no prompt engineering trick gets around it.

At roughly 450 tokens per page of typical prose, and after leaving room for a system prompt and the output itself, the usable window works out to somewhere around 15 to 18 pages. That's a real constraint, not a minor inconvenience, for the exact document types this article is about.

What does that rule out in a single pass?

• Full legal agreements running past 20 pages
• Annual reports or 10-K filings
• Multi-document research or due-diligence packages
• Extended clinical histories spanning multiple visits

Mistral 7B (from version 0.2 onward) offers a 32K context window at a similar parameter scale, and Qwen 2.5 14B reaches 128K, enough to handle roughly 200 pages in one pass. Mistral 7B (from version 0.2 onward) offers a 32K context window at a similar parameter scale. Qwen 2.5 14B reaches 128K, enough to handle roughly 200 pages in one pass. Against those numbers, Gemma 2's 8K ceiling is its clearest structural weakness for anyone working with long documents.

That weakness is exactly why chunking strategies are essential here. They're the only way Gemma 2 handles a real 100-page contract at all.

Three strategies for summarizing documents that exceed the context window

Prompt stuffing is the simplest option: paste the whole document into a single prompt and let the model summarize it in one pass. It only works if the document actually fits inside the 8,192-token window, so realistically, documents under 15 to 18 pages. Short briefs, executive summaries, a single meeting transcript. No orchestration, no extra code, nothing to break.

Map-reduce handles anything longer. Split the document into chunks, optionally with some overlap between them so context doesn't get severed mid-sentence, summarize each chunk on its own (the "map" step), then feed those chunk-summaries back into the model for one final pass that summarizes the summaries (the "reduce" step). A proof-of-concept run on a 120-page document split it into 245 chunks, requiring 246 total calls to the model. The map stage, summarizing every individual chunk, ate up more than 98% of the total runtime. Reduce was fast. Map was the bottleneck, which means running chunks concurrently rather than one after another is the single biggest speed lever available.

Map-reduce trades depth for breadth. It covers the whole document, but relationships between two sections sitting far apart, say, a definition on page 4 that gets modified by a clause on page 90, can get lost, since each chunk gets summarized in isolation. Good fit for long reports where broad coverage takes priority, since each chunk gets summarized in isolation and relationships between distant sections can get lost.

Iterative refinement takes the opposite approach. Summarize the first chunk, then hand the model the running summary alongside the next chunk and ask it to update the summary accordingly, working through the document in sequence. This preserves narrative continuity far better than map-reduce, since the model always has the full running context of everything summarized so far. The cost is speed: there's no parallelizing a sequential process, so a very long document takes proportionally longer to process. This is the right fit for contracts or narratives where a later clause meaningfully qualifies or overrides something stated earlier, the exact structure that trips up map-reduce.

So which one to reach for? Document type and length answer that question directly. Short documents don't need any of this, prompt stuffing handles them. For most private-document work in legal, medical, and financial contexts, iterative refinement tends to do a better job holding onto cross-section relationships. Map-reduce wins when the priority is processing volume and broad coverage, at the cost of tracking how section 12 amends section 3.

Hardware requirements for each Gemma 2 size

Numbers below assume Q4_K_M quantization, a common 4-bit compression format that keeps most of the model's quality while cutting its memory footprint sharply.

Gemma 2 2B needs roughly 1.5 to 2 GB of VRAM, plus some headroom for the KV cache that holds context during generation. That's low enough to run on an ordinary consumer laptop with no dedicated GPU to speak of, which makes it the right fit for an analyst or clinician working somewhere fully air-gapped, no network connection at all. It uses GQA, the same memory-efficient attention design used across the Gemma 2 family.

Gemma 2 9B needs about 6 to 7 GB of VRAM, comfortably inside an 8 GB GPU, something like an RTX 3070 or 4060. For most private-document workflows, this is the sweet spot: strong summarization quality, runs on a single consumer GPU, no data center required. It uses GQA and, notably, generally holds up well at the Q4_K_M quantization level, retaining strong capability after compression. Trained on that 8-trillion-token corpus mentioned earlier.

Gemma 2 27B needs roughly 17 to 18 GB of VRAM, which means a 24 GB GPU, an RTX 3090 or 4090, or an Apple M-series machine with 24 GB or more of unified memory. At full precision it also runs on an NVIDIA A100 80GB, an H100, or a single Google Cloud TPU host. Trained on an enormous corpus of tokens, it delivers the highest factual coverage and the tightest instruction fidelity in the family, worth the extra hardware for high-stakes, regulated document work where accuracy outweighs convenience.

On the software side, a handful of runtimes make local deployment straightforward:

Ollama offers one-command deployment for registry models and supports GGUF format natively

On the framework side, Hugging Face Transformers supports Gemma 2 natively, and JAX, PyTorch, and TensorFlow all support it through Keras 3.0. Teams with an existing ML stack don't need to rebuild anything to bring Gemma 2 in.

Prompt formatting and hallucination control, the two levers that determine output quality

Gemma 2's instruction-tuned models expect an exact control-token format:

<start_of_turn>user
[prompt text]<end_of_turn>
<start_of_turn>model

Generation ends with <end_of_turn><eos>. This isn't a loose convention, it's closer to a hard requirement. Deviate from the template and output quality drops sharply. Deviating from the template causes output quality to drop sharply.

Instruction following is really the metric that matters for summarization work. If a prompt asks for exactly three bullet points, the model should return exactly three bullet points, not two, not five. Structural compliance is as much a measure of quality here as whether the content is accurate. Instruction-following quality varies across open models, and Gemma 2 is solidly capable for its size class; for most summarization tasks, it won't be the deciding factor.

Hallucination control deserves its own attention, because in regulated document work, a fabricated detail is worse than no summary at all. A hallucinated clause in a legal summary or an invented statistic in a financial brief doesn't just fail to help, it actively misleads whoever reads it next. A few concrete levers:

• Larger models generally hallucinate less, and within the Gemma 2 family the larger sizes tend to be more reliable on this front
• Lowering the temperature reduces output variance and cuts down on invented detail
• Explicit grounding instructions, something as direct as "only include information stated in the document", tend to do more work than temperature adjustment alone
• The post-training process included steps aimed at improving grounded output quality, so the model is designed to support this kind of grounding prompt rather than fighting against it

Put those two levers together, correct control-token formatting and an explicit grounding instruction in the prompt, and that combination is close to the minimum viable setup for any production summarization task built on Gemma 2. Skip either one and quality drops in ways that are hard to diagnose after the fact.

Fine-tuning Gemma 2 on domain-specific documents

General-purpose Gemma 2 handles a wide range of document types reasonably well out of the box. But domain-specific language, clinical shorthand, legal boilerplate, financial jargon, and domain-specific document structure, discharge summaries, merger agreements, 10-K sections, benefit from targeted adaptation. That's the case for fine-tuning.

On the license front, the models are available under terms that should be reviewed carefully for commercial fine-tuning and deployment of the resulting fine-tuned models, though anyone moving toward production should confirm the current Gemma Terms of Use against their specific use case before committing engineering time.

Parameter-efficient fine-tuning (PEFT) makes this achievable without retraining the whole model. LoRA and QLoRA adapters can produce strong results on the 9B base using a small fraction of the model's total parameters, a small fraction of the model's total size. QLoRA goes further, making fine-tuning possible on hardware that couldn't even load the full-precision model in the first place.

A concrete published example: fine-tuning google/gemma-2-2b-it on the lavita/ChatDoctor-HealthCareMagic-100k dataset for medical question-answering. That's a useful data point on its own, because it shows even the smallest 2B variant is a viable fine-tuning target for a domain-sensitive task, not just the larger models.

The instruction-tuned variants followed a post-training process that included reinforcement learning from human feedback (RLHF) to keep the model from drifting too far from its base behavior. Teams fine-tuning their own variant can follow that same shape.

And the integration story holds here too: Hugging Face Transformers, JAX, PyTorch, TensorFlow, all supported, so a fine-tuning pipeline slots into infrastructure that's probably already in place.

Fine-tuning on internal documents, done locally, keeps the privacy guarantee intact through the entire lifecycle, not just at inference time. The training data stays inside the environment throughout. That's a meaningful distinction from any fine-tuning workflow that routes through a third-party training service, where the sensitive documents used to adapt the model pass through infrastructure outside anyone's direct control.

Choosing the right Gemma

The decision comes down to three questions: how sensitive is the document, how long is it, and how much hardware is available to throw at the problem.

For laptop-class deployment with no GPU, or an air-gapped environment where installing anything beyond the model itself isn't an option, 2B is the only real choice, and knowledge distillation means it performs better than its size suggests. For the bulk of private-document summarization work, contracts under 20 pages, clinical notes, internal memos, 9B is probably the right default: it fits on a single consumer GPU with 8 GB of VRAM, and its quantization holds up well under compression. Reach for 27B when the stakes are high enough that the extra hardware cost is worth it: regulated filings, high-value legal work, anything where an extra few percentage points of factual reliability changes the risk calculus.

Whichever size gets chosen, the 8,192-token ceiling shapes the whole pipeline design. Documents under roughly 15 to 18 pages can go through prompt stuffing and skip the added complexity entirely. Anything longer needs either map-reduce, for breadth across long reports, or iterative refinement, for documents where later sections change the meaning of earlier ones. There's no universal answer here, only a fit between document type and technique.

What ties every one of these choices together is the same guarantee, regardless of which size or which chunking strategy: nothing leaves the machine. That's the reason Gemma 2 belongs in the conversation for sensitive-document summarization in the first place, and it's a guarantee that holds whether the deployment is a 2B model on a laptop or a fine-tuned 27B model running on a private A100 cluster.