Most "long-term memory" in deployed LLM systems is a single vector index with a similarity search in front of it. That design answers exactly one question — what stored text is most cosine-similar to the current query? — and it answers it without regard to who is asking, when the memory was formed, whether it is still true, or whether it was ever worth keeping. In a demo this is invisible. In a production system where cognitive agents run for months, serve many isolated principals, and act on what they recall, naive RAG fails in ways that are structural, not parametric: no amount of embedding-model upgrades fixes a memory system that has no notion of memory type, memory lifecycle, or memory ownership.
We run a fleet of cognitive personas inside Luca AI Express, our production AI operating system — agents that operate a PERCEIVE→REASON→PLAN→ACT→REFLECT loop over real business domains, day after day. Memory is not a feature bolted onto them; it is the substrate that makes a persona a persona rather than a stateless completion endpoint. This page describes the memory architecture we converged on after the naive design failed us, and the reasoning behind each layer. It is written conceptually — the implementation is proprietary and parts are patent-pending — but the design arguments stand on their own.
The core mistake: treating memory as a retrieval problem
Retrieval is the last step of memory, not the whole of it. Human memory research — and fifty years of cognitive architectures from ACT-R and Soar onward — draws distinctions that turn out to be load-bearing in software:
- Episodic memory: what happened, when, to whom. Time-stamped, autobiographical, high-volume, mostly disposable.
- Semantic memory: what is true. Distilled facts and stable knowledge, decoupled from the episodes that produced them.
- Procedural memory: how to do things. Skills, playbooks, learned action sequences.
A single vector store collapses all three into one undifferentiated pile of chunks. The consequences are predictable. Episodic noise ("the user said hello at 9:14") crowds out semantic signal ("the user's fiscal year ends in March"). Stale facts and their corrections coexist as sibling vectors, and similarity search — which has no concept of supersession — returns whichever one happens to embed closer to today's phrasing. Procedural knowledge, which is best expressed as executable structure rather than prose, gets flattened into text chunks that retrieve poorly and execute not at all.
Our position: the store must be layered before retrieval can be smart. Each layer gets its own schema discipline, its own write path, its own retention policy, and its own retrieval semantics.
The consolidation cycle: wake, attention, sleep
Layers alone are not enough; you need a process that moves information between them. We frame this as a wake/attention/sleep doctrine, deliberately echoing the biological analogy because the analogy pays rent:
WAKE ATTENTION SLEEP
┌───────────────┐ ┌────────────────────┐ ┌─────────────────┐
│ hydrate role, │ │ act; append raw │ │ consolidate: │
│ semantic core,│ ───► │ episodes to the │ ───► │ distill, dedupe,│
│ open threads │ │ episodic log │ │ promote, forget │
└───────────────┘ └────────────────────┘ └────────┬────────┘
▲ │
└───────────────────────────────────────────────────┘Wake reconstitutes the agent: its role definition, its distilled semantic memory, and unresolved items carried over from prior sessions. Crucially, wake loads a bounded, curated working set — not "everything similar to nothing in particular."
Attention is the working phase. The agent acts; raw episodes are appended cheaply and promiscuously to the episodic log. Write-time is not the moment for curation — an agent mid-task has neither the context nor the budget to decide what will matter later.
Sleep is where the real memory work happens, offline and asynchronous. A consolidation pass reads recent episodes and performs the operations similarity search cannot: distilling durable facts into the semantic layer, detecting contradictions with existing semantic entries and resolving them by supersession (the new fact replaces the old, with lineage preserved), promoting repeated successful action patterns toward procedural memory, and marking the bulk of raw episodes for decay.
Separating write-time from consolidation-time is the single highest-leverage decision in the architecture. It converts memory quality from a per-turn inference problem (expensive, latency-bound, poorly contextualized) into a batch reasoning problem (cheap, deliberate, and — importantly — evaluable, because consolidation outputs can be scored and gated before they enter the long-term store). We apply the same evidence-gated pattern here that our deterministic scoring engines use elsewhere in the platform: an AI pass may propose a promotion or a supersession, but it must cite the episodes that justify it, and the gate is checkable.
Ranking is a service, not a query
The second structural fix: retrieval scoring must be pulled out of the storage engine and into a dedicated ranking service with its own model of relevance. Cosine similarity is one feature among several, not the answer. A production ranker for agent memory needs at minimum:
- Recency and decay — episodic memories should lose weight on a schedule; semantic ones mostly should not.
- Layer-aware routing — a "how do I…" query should weight procedural memory; a "what is…" query, semantic; a "what happened…" query, episodic. Query intent classification before retrieval is cheap and pays for itself immediately.
- Provenance and confidence — a fact confirmed across many consolidation passes outranks one distilled from a single ambiguous episode.
- Usage feedback — memories that, when retrieved, contributed to successful task completions (a signal the REFLECT phase can emit) should rise; memories repeatedly retrieved and ignored should sink.
Making ranking a versioned service has a second-order benefit that we consider non-negotiable: reproducibility. When an agent misbehaves, "what did it remember and why" must be answerable. A deterministic, versioned ranker with logged feature values turns a memory-related incident from vibes into a diff. This mirrors how we version our scoring engines generally — behavior changes ship as ranker versions, with the change and its rationale recorded, not as silent drift.
Embedding hygiene
Embeddings are infrastructure, and like all infrastructure they rot without maintenance discipline. The failure modes we treat as first-class:
Model-version skew. Vectors from different embedding model versions are not comparable, and mixed-version indexes degrade silently — recall drops with no error anywhere. Every vector must carry its embedding-model version, and a model upgrade is a migration event with dual-write or full re-embed, never an in-place swap.
Chunking as a semantic decision. What you embed determines what you can recall. Embedding raw conversational turns yields vectors dominated by phatic noise. We embed consolidated artifacts — distilled facts, episode summaries, procedure descriptions — in preference to raw text wherever the layer allows it. The sleep phase, conveniently, is exactly where such artifacts are produced.
Query/document asymmetry. Queries and memories are different linguistic objects; techniques like instruction-prefixed embeddings or hypothetical-document expansion (HyDE-style) matter more in memory systems than in document search, because memory entries are short, telegraphic, and written by a machine for a machine.
Index audit. Periodically sample the index: orphaned vectors whose source rows were superseded, duplicate near-identical entries from pre-dedup eras, embedding outliers from ingestion bugs. Nobody notices index rot until recall quality has already cost you.
Forgetting is a feature
The instinct in engineering is to keep everything — storage is cheap. For agent memory this instinct is wrong, for three reasons.
First, retrieval quality: every retained low-value memory is a candidate distractor in every future top-k. An index that only grows has monotonically worsening precision. Forgetting is how you spend storage to buy signal-to-noise.
Second, truth maintenance: facts change. A memory system without supersession semantics will eventually retrieve a stale fact with full confidence, and an agent will act on it. We treat semantic memory as versioned assertions: corrections don't sit beside the facts they correct, they replace them, with the old version retained in lineage for audit but excluded from retrieval.
Third, governance: real deployments face retention obligations and deletion rights. "Delete this user's data" must be a supported operation across all layers — including derived artifacts. This is genuinely hard: a semantic fact distilled from an episode inherits the episode's data-governance obligations, so consolidation must propagate provenance, or deletion becomes unimplementable. Design this in on day one; retrofitting provenance onto an existing memory corpus is close to a rewrite.
Mechanically, forgetting is graduated: decay scores that demote before they delete, consolidation that replaces a hundred episodes with one summary (lossy compression is forgetting, applied usefully), and hard deletion driven by policy. The sleep phase owns all three.
Isolation is a property of the store, not the prompt
Multi-tenant agent memory has a threat model. The naive approach — one shared index, a tenant-id metadata filter applied at query time — makes isolation depend on every query path remembering to apply the filter. One forgotten filter in one code path is a cross-tenant memory leak, and unlike a web-app data leak, this one gets paraphrased into an agent's fluent output, making it hard to even detect.
Our rule: isolation is enforced below the retrieval API, not within it. Per-principal memory partitions are structural — a retrieval call executes inside a scope from which other principals' memories are not merely filtered but unreachable. Shared organizational knowledge lives in explicitly designated shared layers with their own access semantics, and the ranking service never blends scopes implicitly. The corollary for consolidation: a sleep pass runs within a principal's scope; cross-principal distillation ("many users asked X") is a separate, deliberately designed aggregation pipeline with its own privacy review, never an accident of a shared index.
Failure modes we have paid for
Candidly, the lessons that shaped this design:
- Consolidation hallucination. A distillation pass can confidently write a "fact" the episodes don't support. This is why consolidation outputs are evidence-gated — the pass must cite supporting episodes, and unsupported promotions are rejected. Without the gate, the semantic layer becomes a slow-cooking hallucination amplifier: wrong facts get retrieved, reinforce themselves in new episodes, and reconsolidate.
- Feedback-loop ossification. Usage-weighted ranking can lock in early memories: retrieved because ranked high, ranked high because retrieved. We damp this with exploration in ranking and by capping usage signal relative to provenance signal.
- Over-forgetting. Aggressive decay once erased context an agent needed weeks later. Demote-then-delete, with a quarantine window, is the safety net.
- The memory/instruction boundary. Memories injected into context are data, but models treat fluent imperative text as instructions. Consolidated memories that happen to contain directive language can steer behavior. Consolidation therefore normalizes memories into declarative, attributed forms — a mitigation, not a solution; this remains an open problem for the field.
Every one of these lessons lives in our Codex — the institutional knowledge system through which every change to the platform ships with an executable blueprint/playbook/runbook triple. The memory subsystem's own evolution is browsable as a dependency and knowledge graph at ticket.lucaexpress.com, which is fitting: the Codex is, in effect, the organization's semantic and procedural memory, built on the same conviction that knowledge you cannot retrieve, audit, and supersede is knowledge you do not have.
Where this is heading
Three directions occupy our current research. First, tightening the loop between memory and our Model Forge pipeline: consolidated procedural memory is close to fine-tuning data, and we are exploring when a repeatedly-executed playbook should stop being retrieved context and start being distilled into the weights of a small specialized model — memory consolidation continued by other means. Second, richer forgetting: learned per-layer decay policies evaluated against downstream task success rather than hand-set half-lives. Third, cross-agent memory economics — what a fleet of isolated personas can safely share, and through what governed aggregation channel, without eroding the isolation guarantees above.
The architecture summarized here — layered stores, offline evidence-gated consolidation, a versioned ranking service, hygiene-managed embeddings, graduated forgetting, and structural isolation — is more machinery than a vector index and a prompt. It is also, in our experience, the minimum viable shape for memory you can operate, audit, and trust in production.
Gus IT Research Institution takes on external research and engineering engagements in agent memory, small-model pipelines, and AI systems architecture — reach out through isalabresearch.com.