Persistent Context and Memory in Local Open-Weight Models

Local, open-weight models don't remember anything between sessions. Tell a coding assistant running on your own machine that you use pytest, and it'll write unittest boilerplate the next time you open a terminal. Mention that you want type hints on every function, and by tomorrow it's back to an untyped version of the same language like the conversation never happened. That's not a bug someone forgot to patch. It's the architecture working exactly as designed, and understanding why is the first step toward fixing it.

An LLM processes whatever's in its context window, produces a response, and then the slate wipes clean. No hidden state carries over. No file gets written to disk unless something outside the model explicitly does that. Cloud providers have gotten good at papering over this with memory layers running on their own servers. But local models don't get that convenience by default, and the whole reason someone chooses to run open-weight models locally is usually privacy: keep the data off remote servers, keep control of the hardware. Bolting on a cloud memory service defeats the purpose of going local in the first place.

So what would actual persistent context look like here? Facts stored somewhere outside the context window. The system finds what's relevant rather than dumping everything back in. Durable across restarts. And running entirely on hardware you control. Four different architectural approaches try to deliver this, and each one trades something away to get there. All four are worth walking through, because the right choice depends heavily on what you're building.

Why stuffing context into a larger window is only a partial answer

The simplest fix looks obvious at first: just make the window bigger. If a model can hold a million tokens instead of eight thousand, why not just paste the entire conversation history back in every time?

Open-weight models have gotten genuinely enormous in this respect. Qwen3-30B-A3B-Thinking-2507 handles 262,144 tokens natively and stretches to a million with extension, all while running as a mixture-of-experts model with only a small fraction of its total parameters active at any time. GLM-5.2 from Z.ai pushes even further: a very large parameter count, a 1,048,576-token window, and a sparse attention scheme called DeepSeek Sparse Attention that reduces per-token compute substantially. Its licensing terms are worth checking before commercial use. Jamba 1.5 400B Large mixes Transformer and Mamba components for a 256K window. Llama 4 Scout goes further still, claiming a context window of up to ten million tokens.

These numbers matter for certain jobs, like summarizing a long document or holding a big codebase in view during a single session. But none of it amounts to memory in any meaningful sense.

A bigger window is just a bigger piece of paper. Nothing about a longer prompt enforces the things memory actually needs, like deciding what to keep, what to let fade, or how to reconcile a new fact with an old one. Context compaction still kicks in during long sessions, and token limits still apply, so a preference you mentioned three thousand tokens ago can get quietly dropped even inside a supposedly huge window.

The common workaround, a PREFERENCES.md file loaded at startup, or a long system prompt, or some CONTEXT.md a developer maintains by hand, all share the same flaw. They live inside the window. Restart the session, and unless something outside the model reloads them, they're gone. Worse, with genuinely open-weight models like Qwen2.5-1M, the training data behind these long-context capabilities stays proprietary, so it's hard to know exactly how the model behaves at the far edges of that window.

A bigger window buys breathing room. It doesn't buy a memory system. That gap is exactly what the next three approaches try to close.

How RAG turns an external document store into a memory layer

Retrieval-augmented generation splits memory into two pieces: what's baked into the model's weights from training (parametric memory), and what lives in an external index that can be updated without retraining anything (non-parametric memory). For a local setup, that second piece is the interesting one.

Here's the mechanic. Facts, notes, or documents get embedded into vectors and stored in a database. When a query comes in, the system searches that database for the closest semantic matches and pulls them into the context window before the model generates a response. The model never has to hold everything in its head. It just has to know how to ask.

Local tooling for this is mature. ChromaDB handles automatic text embedding through Sentence Transformers (the default model, all-MiniLM-L6-v2, produces 384-dimension vectors), runs approximate nearest neighbor search via HNSW, supports metadata filtering, and persists everything to local storage, sometimes as a lightweight SQLite3 database. Qdrant handles larger document collections and shows up often in production local stacks. Both are genuinely solid pieces of infrastructure, and the market around them backs that up: vector databases, the core piece of RAG infrastructure, were valued at $2.2 billion in 2024, with projected growth at a strong double-digit annual rate through 2034.

So where does this fall short of real memory? A few places, and they're structural, not incidental.

Retrieval in a standard RAG setup is read-only. Facts sit in the store unchanged, and if two entries contradict each other, there's no resolution step, just whichever one scores higher on similarity at query time. There's no decay mechanism either: a stale fact from six months ago competes on equal footing with something added yesterday. And crucially, the store has no concept of relationships between facts. It can't represent that one entry supersedes another, or that two facts are actually about the same underlying thing.

Agent frameworks try to patch this with rolling summaries or pinned "memories" or hand-built slots for specific fact types. But squint at those workarounds and they're just strings inserted into a prompt. That's not a memory architecture. It's the same flat-file problem from the last section, dressed up with a vector search step in front of it. Some AI assistants, like Vellum, a memory-powered AI assistant that learns user preferences and takes autonomous actions across tools such as email and Slack, address this by managing memory at the application layer rather than relying on retrieval alone.

Dedicated memory frameworks: how Mem0, Letta, and their peers go beyond RAG

The gap RAG leaves behind, no conflict resolution, no decay, no structure, is exactly what a newer generation of memory frameworks set out to fix. The core idea: put an LLM in the loop before anything hits the vector store, so the system decides what's worth keeping, how to categorize it, and whether it clashes with something already stored.

Mem0 is probably the clearest example. It's open source under Apache 2.0, built by Taranjeet Singh and Deshraj Yadav, who'd previously worked on Embedchain, an open-source RAG framework that racked up more than two million downloads. Mem0 itself passed 37,000 GitHub stars by mid-2025 and raised a substantial funding round in October of that year.

Architecturally, it's a hybrid: a vector database for semantic similarity, a graph database for modeling relationships between facts, and a key-value store for fast direct lookups. To be precise here, Mem0 isn't a database itself. It's an orchestration layer that runs Qdrant by default for vector search and SQLite for history tracking. Self-hosting means running Mem0 alongside those services, not dropping in a single binary.

For a fully local setup, Mem0 connects to Ollama, which means the whole stack, model inference, fact extraction, embedding, storage, can run offline. One documented pattern from Mem0's own blog in April 2026 lays out the pipeline: Ollama runs the model locally and handles intent detection, code generation, and memory classification; Mem0's open-source layer extracts facts (calling the LLM to pull structured facts, then the embedder to vectorize them); Qdrant persists everything to disk, so it survives a restart.

On the LOCOMO benchmark, an evaluation covering a range of question types, Mem0 outperformed the memory systems it was tested against, scoring 26% better on an LLM-as-a-judge metric compared to OpenAI's approach, while cutting p95 latency by 91% and token costs by more than 90% relative to just stuffing full context into the window.

Letta takes a different architectural bet. It started life as MemGPT, a research project out of UC Berkeley, and has since evolved into Letta Code, a memory-first coding agent backed by a venture investor with a modest seed round. Instead of a flat store, Letta organizes memory into tiers: core memory that sits directly in context, archival memory (a relational database with a vector-search extension when self-hosted, or TurboPuffer on Letta's managed cloud), and a tier for retrieving past conversation. It's the same basic idea as an operating system managing RAM versus disk, applied to an LLM's memory.

The interesting part is who's in charge. The agent itself decides when to read from or write to each tier, using tool calls, acting as its own memory controller rather than a developer hand-coding the retrieval logic. Letta is model-agnostic too, supporting OpenAI, Anthropic, Mistral, and local models without locking a user into one vendor. Letta Code scored 42.5% on Terminal-Bench, the top result among model-agnostic open-source agents, and hit 74.0% accuracy on LoCoMo using GPT-4o mini. The catch: Letta needs persistent infrastructure, a database and typically Docker, running alongside it. That's real deployment overhead for someone who just wants a local setup with minimal moving parts.

Beyond those two, the ecosystem has filled in quickly. Graphiti, built by the Zep team and open source under Apache 2.0, constructs temporally aware knowledge graphs that track not just facts but how they change over time, which suits agents operating in domains where information has a shelf life. Zep CE is the self-hosted community edition of Zep's memory server, combining temporal graph storage with vector search. LangMem, Memary, and Cognee round out the current field of open-source memory layers as of mid-2026.

Then there's AnythingLLM from Mintplex Labs, free and open source, which takes a different approach entirely: it's a full application that connects to local model runners like LM Studio or Ollama, with memory built in rather than bolted on. A background process runs periodically through recent chats, pulls out facts as single-sentence memories, and re-injects them into the system prompt at chat time. Memories get split into two scopes, workspace-specific and global across all workspaces, and users can also add memories by hand. There's a practical threshold too: a workspace needs at least five unprocessed chats before automatic extraction kicks in, so short exchanges get skipped rather than cluttering the store.

Is this architecturally cleaner than a PREFERENCES.md file? Not really, it's still injection into a system prompt under the hood. But the automatic extraction removes the manual maintenance burden, and the workspace/global split adds a layer of structure the flat-file approach never had.

So how do you actually choose between these? It mostly comes down to three questions. Can the setup tolerate running a database and Docker, or does it need to stay lean? Does the use case need temporal or graph-style reasoning, or is flat semantic retrieval good enough? And should memory management be fully agent-autonomous, the way Letta hands control to the model, or handled through an orchestration layer but developer-controlled, which is closer to how Mem0 operates?

State space models as an architectural alternative to KV-cache memory

Every answer so far has treated memory as something bolted on from the outside. But why does the transformer architecture need external memory help in the first place? The answer traces back to how transformers handle attention internally, and it turns out that mechanism has a cost problem of its own.

Every token a transformer generates requires attending back over the full sequence of previous tokens, and the key-value cache that makes this efficient grows with sequence length, pushing memory demands up linearly or worse. Compute for self-attention itself scales quadratically as the sequence gets longer. That's a real bottleneck, independent of any external memory system.

State space models, Mamba being the best-known example, take a different bet. Instead of storing and attending back over every previous token, Mamba keeps a compact hidden state and updates it as new tokens arrive, giving it linear, O(N) scaling instead of quadratic. During inference, each new output comes from the previous state directly, no KV cache, no recomputing the whole sequence history from scratch. The state size per layer stays fixed no matter how long the sequence runs.

That's the appeal. But it comes with a real cost: a fixed-size state is, by definition, a compression of everything that came before it, and compression loses detail. Mamba models don't consistently match transformer accuracy across every task type. Transformers still win where a task needs precise, fine-grained attention over specific tokens. Mamba tends to do better where long-range dependencies matter more than pinpoint recall of any single detail, but it also shows a known tendency to forget information that's far back in the sequence, precisely because that fixed state can't hold everything at full resolution forever.

The research line continues to move. Mamba-3, described in a 2026 paper, builds on Mamba-2 and Gated DeltaNet, and its ideas already appear inside larger hybrid models. Jamba 1.5 400B Large is the clearest example of where the field is landing practically: Transformer layers where precision matters, Mamba layers where raw efficiency over long range matters more.

Though, SSMs don't replace the memory frameworks discussed earlier. They just relocate the problem. Instead of paying a cost in external retrieval latency, the cost shows up as internal state compression. An agent that needs durable, cross-session facts, the kind Mem0 or Letta are built to hold onto, still needs an external store no matter how efficient the underlying architecture gets at handling a single long sequence.

What stored memories get wrong when they're not managed carefully

Every framework covered so far assumes that once something's stored, it's basically fine to bring back up whenever it seems relevant. That assumption deserves more scrutiny than it usually gets.

A study called PersistBench, run across the SPAR, Cambridge, and CISPA research groups and covering 18 frontier and open-weight models, tested exactly this. It looked at two specific failure patterns. The first, cross-domain leakage, is what happens when a memory from one context bleeds into an unrelated one, say, something learned in one conversation appears uninvited in an unrelated one. The second, memory-induced sycophancy, is when a stored belief about the user starts biasing the model toward agreement rather than accuracy, effectively prioritizing "stay consistent with what this person already believes" over "give the correct answer."

The results were rough. Median failure rate on cross-domain leakage samples landed at 53%. On sycophancy, most models failed more than 90% of the time, with a median of 97%, a number the researchers themselves flagged as surprisingly high. Samples involving personal identity, where the user's self-description was stored as memory, turned out to be the most prone to sycophancy, which is a fairly direct path toward the model becoming an echo chamber across sessions rather than a useful second opinion.

A separate benchmark, BenchPreS, from researchers at Yonsei University and LG AI Research, probed something adjacent but distinct: can a model apply a stored preference only where it actually belongs, rather than everywhere, indiscriminately? That's called context-aware preference selectivity, and the failure mode is almost comic when it occurs. One example from the paper: a user's memory profile includes a preference for jokes, emojis, and playful tone. Ask that same model to draft a formal letter to an IRS agent, and it comes back written like a school newsletter, signed "Joker."

Both findings point at the same structural gap. Storing a fact durably and retrieving it semantically, the two things every framework in this piece is built to do well, say nothing about whether that fact should apply to the situation at hand. Building memory that persists is one problem. Building memory that knows when to stay quiet is a different, and arguably harder, one.