Measuring Memory

The result that reframes the rest

Agent memory systems are usually scored end-to-end: feed a long history, ask a question, check the answer. It is an intuitive protocol and it hides a structural problem. An end-to-end score is the product of at least two independent components — whether the right context was retrieved, and whether the model read it correctly — and the published number attributes the result to the memory system alone.

We measured how much of a memory benchmark score is actually attributable to memory. On identical retrieval, holding the judge frozen, swapping only the reader model moved our LongMemEval score by 16.4 percentage points. That single result reframes every number in this article, including our own, so we report it first.

What Velixar is

Velixar is a persistent, cross-session, cross-agent memory substrate. It is not a model and does not train one; it sits underneath agent runtimes and protocols (MCP today) and holds what a system of agents collectively knows.

Three design principles matter for interpreting the results below. We state them as commitments rather than mechanisms:

Memory is a belief state, not a filing cabinet

What the system holds must remain coherent, attributable, and revisable over time. Retrieval quality is necessary but not sufficient; memory that returns the right document while holding an incoherent picture of the world is doing only half of its job.

Authority is earned, not assumed

Most memory systems have exactly one trust level: written means true. Velixar has more than one. Newly written memories do not automatically carry the same weight as validated ones; trust is something a memory earns over time. Retrieval, ranking, and confidence all take that standing into account.

Contested facts are preserved, not resolved

When two memories make incompatible claims, the system detects the conflict, records it with reasoning, and keeps both claims visible rather than silently picking a winner. A fact changing over time is modeled separately from a genuine contradiction, and superseded values are retained rather than deleted.

Memory volumes are isolated per tenant at every storage tier and every index — enforced and canary-tested rather than assumed.

Our number, under the tightened protocol

80.8% ± 3.0

Two things about this number are more important than the number.

It is lower than the figure we previously published, and we revised it anyway. Version 1 of this page reported 82.1 ±1.8 under a protocol that allowed the same model to read and judge, and ran fewer seeds. Enforcing reader ≠ judge and running the full 120×3 grid produced 80.8 ±3.0. The direction of the correction is not a regression in the product — retrieval was identical across both protocols — it is the measurement getting harder to flatter. A benchmark protocol that only ever moves your number up is not a protocol; it is marketing.

Its error bar widened, from ±1.8 to ±3.0, because more seeds revealed more variance than the earlier run had sampled. We report the wider bar for the same reason we report the lower mean.

Token efficiency

Measured with real tokenizer counts (o200k_base) across all 120 questions:

Metric	Tokens
Full-context (entire ~48-session history)	103,442
Velixar retrieved (what the reader saw)	16,274
Saved	87,168

~84% fewer tokens than stuffing the whole history — about 10.5M tokens saved across the run, at 82% accuracy. At topk=25 the system returned only what cleared the relevance threshold (~10 sessions per question on average), not a fixed pad of context. A full-context comparison arm with the same reader is underway to state this at matched accuracy.

The reader experiment

The 16.4-point result, in brief: identical retrieved context, identical judge, two readers. Everything about the memory layer was held fixed; only the model reading its output changed. The score moved by more than the published gap between most competing memory systems.

We have since run this to exhaustion — every pairing of seven readers and seven judges, 49 cells — in The Jury Nobody Names: 22.2 points of spread from reader choice, 6.8 from judge choice, +0.9 mean self-preference. And we have measured the half of the blend that is ours, with no model in the loop, in Retrieved, Not Read: 98.3% retrieval recall@25 on LoCoMo, 97.4% for complete evidence sets.

One reader-side result worth restating here because it breaks an intuition: reader strength is not monotonic with model generation. A newer model read worse than its predecessor in our grid (71.2% vs 78.8%, judge-averaged). Reader quality has to be measured, not assumed from a release date.

On comparisons we are not making

An earlier revision of this page placed our score alongside published figures for other memory systems. We have removed those comparisons. Our own reader experiment is the reason: published end-to-end numbers were produced with different readers, different judges, and undisclosed retrieval budgets, and the spread attributable to those choices exceeds the gaps between the systems. The only comparison that carries information is both systems in one harness with one reader and one judge. That is work we have not done, so it is a claim we do not make. Our harness runs through the standard client path against the shipping product; anyone who wants the shared-harness comparison can run it.

What happened to GPT-4o parity

Version 1 of this page said a GPT-4o reader-parity run was in progress, to check our numbers against the reader most commonly used in published results. That run is cancelled: the model can no longer be deployed on our infrastructure — all available versions are deprecated or deprecating on Azure, and direct API access is rate-limited to unusability. We could not run it, so we are saying so rather than leaving the promise standing.

The underlying question — how much does the reader move the number — is answered more completely by the 7×7 grid than a single parity run would have answered it.

Reliability engineering

Memory failures are not like software failures. A crashed service is loud, bounded, and recoverable. A wrong memory is silent, persistent, and compounding — every retrieval of it can seed further state, and embeddings and consolidated memories are frequently irreversible.

We run every change through a structured review that treats this asymmetry explicitly: candidate failure modes are scored on likelihood × severity × detection difficulty, with detection difficulty weighted heaviest for memory and graph changes. Each identified risk carries a defined invariant, an adversarial probe, a production detection signal, a containment path, and a recovery path. Isolation and correctness invariants are asserted continuously in production, not assumed to hold.

Protocol, stated

LongMemEval, 120 questions, 3 seeds, retrieval budget k=25, reader and judge named and distinct, isolation enforced per conversation and asserted at answer time. Retrieval budget, sample size, seeds, reader, and judge are stated because a benchmark result that omits them cannot be checked — including by us, which is how the revision above happened.

@techreport{velixar-measuring-memory-2026,
  title       = {Measuring Memory},
  author      = {Velixar AI},
  year        = {2026},
  institution = {Velixar},
  url         = {https://velixarai.com/research/measuring-memory}
}