Building a Self-Hosted Memory Layer Around Open-Weight Models for Lasting Work Context

A persistent memory AI assistant for knowledge workers needs more than open-weight models or a large context window. It needs a self-hosted layer that preserves project state, preferences, and relationships between sessions. Building that layer means choosing memory mechanisms, a deployment location, and infrastructure that work together, because their limits decide what useful work context can be kept.

Why every LLM conversation starts from zero, and why that is now a solvable infrastructure problem

Re-explaining a project at the start of every conversation turns repeated work into repeated setup. Large language models are stateless by design, so an application must preserve and restore anything needed from a previous session.

The distinction is between retrieval and persistence. RAG fetches relevant document chunks at query time, but does not itself maintain the evolving state of a working relationship. A memory layer stores that context across sessions and recalls it on demand. Confusing the two produces a system that can find project documents without remembering where the project stands.

Open-weight models have reached production quality, making this an infrastructure decision. The European Commission states that enforcement powers under the EU AI Act apply from August 2, 2026. Third-party API calls can send data across borders or expose it for use as training data, which can affect what is required under HIPAA or EU AI Act. Downloadable weights do not make a managed endpoint sovereign.

What "self-hosted memory" actually means architecturally before any framework is chosen

Document facts, conversational coherence, and persistent work context need different treatment. Document facts belong in a retrievable corpus, while recent messages preserve tone and the nature of an exchange better than isolated facts extracted from its history. Preferences, project state, developer identity, and learned relationships form a third category. That is the dedicated memory layer’s distinct role.

Self-hosting means controlling more than the model weights. The inference server, vector or graph memory store, and observability stack must also run within the organization’s controlled infrastructure. Traces can contain prompts, retrieved chunks, tool inputs, outputs, and user identifiers. Sending those traces to an external monitoring platform exports sensitive context even when inference stays private. The architectural boundary follows the data.

How can an AI assistant remember projects and preferences between sessions? The three memory mechanisms

A large context window can hold a lot of the material needed for work. On its own, it cannot keep that material between sessions. GLM-5.2 and DeepSeek-V4-Pro offer million-token context, but accepted input length is not the same as reliable attention across that input. Multi-fact recall degrades, particularly in the middle of the prompt. Reprocessing the entire corpus for every question also makes long context expensive, with costs typically tens-to-hundredfold above RAG on enterprise workloads against large corpora.

RAG addresses that cost by retrieving relevant material. It suits dynamic document corpora with high query volume. Its limit appears when the task changes from finding evidence to sustaining a conversation. Retrieving individual facts from dialogue does not reliably preserve tone or interaction character, so recent messages still belong directly in context.

A dedicated memory layer supplies what both approaches lack. It preserves work state and uses that state to narrow retrieval before document search begins. The 2026 consensus pattern therefore combines the mechanisms in sequence. Memory filters and supplies persistent context, RAG retrieves a broad candidate set with a generous top-k, and the full candidates pass unchunked into a large context window. This is “retrieve wide, then read long,” with each layer compensating for another’s limits.

Where inference runs and why that decision constrains everything else in the stack

Hardware sizing starts with weights, but cannot end there. Required VRAM includes weight memory, KV cache, and serving-framework overhead. Weight memory follows parameter count multiplied by bytes per parameter, with FP16 using 2 bytes, FP8 using 1 byte, and AWQ INT4 using 0.5 bytes. KV cache grows with context length, batch size, layer count, and hidden dimension. Long context with moderate batches can add tens of gigabytes on a Llama-style architecture.

Mixture-of-experts models introduce another sizing trap. All experts must be loaded, so total parameters determine weight storage even when only a smaller slice activates per token. Frameworks such as vLLM, SGLang, and TGI require an additional buffer. A model that fits at short context may therefore be unable to handle the planned memory workload when it is under load.

At the workstation tier, a mid-sized model quantized to AWQ INT4 can fit available consumer GPU memory. Compression is widening that option. PrismML reports that Ternary Bonsai 2 reduces a large Qwen3.8 model’s memory footprint roughly ninefold while retaining a high share of aggregate benchmark performance. At the server tier, hardware terms also constrain deployment. NVIDIA’s consumer graphics software license technically prohibits RTX 4090 data-center production serving, while L40S is the enterprise-certified alternative.

An on-premises server gives full sovereignty at the highest upfront cost, while sovereignty in a private cloud depends on the contract terms and how data residency is set up. A managed endpoint running open weights still introduces third-party dependencies. Utilization changes the economic calculation too. At low GPU utilization, per-token cost rises sharply, making traffic forecasting inseparable from the deployment choice.

Which open-weight models are viable inference engines for a persistent memory stack

Apache 2.0 and MIT are the safest licenses to start with for commercial self-hosting. Qwen3 is a general-purpose candidate under Apache 2.0, with reasoning, coding, multilingual capability, and sizes spanning different deployment tiers. Phi-4-mini offers an MIT-licensed, long-context option for modest machines. Gemma 3 mid-size variants suit single-GPU multimodal work, but include additional license conditions rather than Apache 2.0 terms.

DeepSeek-V4 Flash and Pro have MIT licensing and million-token context for work that needs broad retrieval followed by long reading. Their total parameter footprint remains the hardware constraint. Devstral instead maps to agentic software engineering, with Apache 2.0 for Small variants and modified MIT for Devstral 2 large. Mistral Small 4 and Mistral Large 3 offer an Apache 2.0 path for EU-sovereign deployments, supported by a European vendor and an air-gapped deployment track record.

The serving engine must fit the same deployment tier. vLLM with AWQ or FP8 is the production-oriented option for CUDA throughput, while GGUF through llama.cpp supports CPU offload and local development. SGLang and TGI offer other production batching and throughput profiles. Local runners such as Ollama give access to serving engines through APIs that work with another vendor’s interface. They simplify access, but do not supply persistent memory themselves.

The four self-hostable memory frameworks and the real tradeoffs each one demands in production

Mem0 offers the simplest self-hosted route when the main requirement is extracting conversational facts and preferences. Its Apache 2.0 framework can run through Docker, has Python and JavaScript SDKs, and supports vector storage with the option to add graph storage. Migration from managed hosting needs attention, however. Self-hosted and managed endpoints use different client classes, Memory and MemoryClient, with some behavioral differences.

Graphiti, the Apache 2.0 framework associated with Zep, is the stronger fit when facts change over time. It combines vector search with entity and fact graphs. Validity windows on fact edges distinguish when something was true from when it was superseded, which vector-only storage cannot match. The tradeoff is Neo4j or FalkorDB operational overhead. Zep’s vendor-published LongMemEval result also diverges meaningfully from an independent comparison, so temporal structure should not be mistaken for independently established recall quality.

Letta, formerly MemGPT, puts context management inside the agent architecture. Its Apache 2.0 platform separates in-context memory from out-of-context storage, allowing agents to manage what remains active and what gets archived. This suits multi-agent pipelines with parallel agents exchanging memory. Teams adopting it are choosing agent-managed context.

Cognee is the publisher’s own product, included here for its documented capabilities and named production deployments. Its ECL pipeline, Extract, Cognify, Load, classifies documents, extracts entities and relationships through an LLM, generates summaries, and commits graph edges and vector embeddings in one operation. Hybrid retrieval combines vector similarity with graph traversal for multi-hop reasoning. With MCP integration, multiple agents, including local Llama instances, can share read and write access. It also supports on-premises multi-tenancy and data residency requirements.

Cognee’s production uses give that architecture concrete grounding. Bayer uses it for scientific research memory supporting hypothesis generation, while the University of Wyoming uses it for policy analysis with page-level provenance. These deployments support its inclusion for graph-structured knowledge work. Its memify refinement stage prunes stale nodes, reweights edges from usage signals, and adds derived facts. That is a partial response to accumulation.

Three failure modes that undermine an otherwise well-designed self-hosted memory stack

The first failure mode is an incomplete observability perimeter. Private inference does not compensate for external trace routing. To avoid that leak, teams should include observability in the self-hosted boundary from the start. Instrumenting memory reads and writes locally keeps operational inspection within the same boundary as the context being inspected.

The second failure mode is memory poisoning through ungoverned accumulation. Conflicting facts, superseded project context, and stale information can degrade retrieval even when storage works correctly. Persistence makes this problem structural. Graphiti’s temporal edges provide the most structurally complete available approach, but engineering teams still have to design invalidation logic. No leading framework has published production-grade invalidation covering the full range of contradictions, superseded facts, and time-bounded relevance.

The third failure mode is treating benchmark claims as production evidence. LongMemEval vendor figures and independent evaluations can diverge substantially, and no framework has a clean independent audit covering multi-session recall, contradictory facts, and high-volume writes under production conditions. Teams should test recall on work like their own before they commit.

Would a managed platform remove these problems entirely? It removes operational burden, but reintroduces dependencies around residency, training exposure, cross-border transfers, and vendor pricing control. Those dependencies cannot be engineered away within the customer’s stack. Self-hosting leaves real, unresolved engineering work, but it also gives the team control over where that work happens and how it is inspected.

How the three decisions compound into a concrete starting architecture

Choosing “retrieve wide, then read long” establishes context requirements that drive KV-cache sizing. That sizing determines feasible hardware and location. Location, in turn, constrains which memory and observability services can run without external dependencies. Choosing Graphiti also means budgeting for its graph database alongside its retrieval code. Otherwise, a dependency the team missed can force part of a stack meant to be sovereign into a managed service.

For a single-team workstation or small private server, a concrete starting point is a small Qwen3 model quantized to AWQ INT4 and served through vLLM behind a local API. Mem0 through Docker suits straightforward facts and preferences, while Cognee suits teams needing graph-structured hybrid retrieval and MCP from the outset. Logs should stay local, with memory reads and writes instrumented. Explicit TTL or supersession logic belongs in the initial design. It is not a later optimization.

For team-scale private cloud, Mistral Small 4 with its 256K context offers an Apache 2.0 option for an EU-sovereign path, served through vLLM with FP8 on L40S or H100-class hardware sized for the workload. DeepSeek-V4 Flash is an MIT-licensed alternative where million-token context justifies the total-parameter memory requirement. Cognee fits graph-native hybrid retrieval, Graphiti fits temporal validity, and Letta fits self-managed multi-agent context. Traces remain inside the compliance boundary.

The assistant should bring up learned preferences and project context without making people explain them again. Human decision authority stays intact. An evaluation or workflow layer above the stack adds value when it makes stored memory visible and lets the team inspect, audit, and override it.