We ran one memory system through seven reader models and seven judges — all 49 combinations. The scores say more about the industry's benchmarks than about any memory system.
Velixar AI
We ran an experiment that any memory vendor could run and, as far as we can tell, none
has published.
Take one memory system — ours. Hold everything constant: the same stored history, the
same retrieved context, the same 120 benchmark questions. Then change only the model
that reads what the memory layer returns, seven times over. Then grade every set of
answers with seven different judge models — all 49 combinations.
Every cell is a measured score: one memory system, seven readers (rows) × seven judges (columns), 49 real cells on LongMemEval (n=120, k=25). Rows fall 22.2 points top to bottom (reader spread); columns vary 6.8 points (judge spread); the outlined diagonal is each model grading its own answers. Means exclude the diagonal.
58.3% → 80.6%
Accuracy of one identical memory system, by reader model
Nothing changed but which model did the reading. A 22.2-point spread — wider than the gap between most competing memory systems' published numbers.
Same memory, same retrieval, same questions — only the reader changes. A 22.2-point spread from Llama-3.3-70B (58.3%) to gpt-5.4 (80.6%). Full 0–100 axis; LongMemEval, n=120, k=25.
Which means that when two vendors report scores 15 points apart, you have learned almost
nothing. The systems could be identical underneath. The ranking could be reversed.
Unless both name the reader model, the judge model, and the retrieval budget, an
end-to-end memory benchmark score is a joint measurement of two products — one of which
the vendor didn't build.
The number that's actually ours
Velixar is a retrieval and memory layer. The model that consumes what we return is the
customer's choice — swapped freely, priced independently, improving on someone else's
schedule. So we report the number that measures our layer and only our layer.
LoCoMo, the standard long-conversation memory benchmark, labels which session of a
conversation holds the evidence for each question. That makes retrieval scoring
deterministic — did the labelled evidence come back, or not. No reader, no judge, no
prompt, nothing to drift when a model gets deprecated.
This is a retrieval figure, not an answer-accuracy figure. It is not comparable to any published end-to-end score, including numbers other memory systems report — those measure something else. The complete evidence set, for questions needing several sessions, came back for 97.4%.
The part that's yours
That 98.3% is the number that stays. It is measured with no model in the loop, so it does not move when the reader is swapped or deprecated — and the institutional knowledge behind it only compounds as your system learns more. The reader is rented and replaced; the memory is owned and accumulates. That is the part that lasts.
The number a buyer should ask for
Retrieval quality is a property of the layer. Its value only shows up against the
alternative, so we measured that too — the same reader, the same questions, with memory
and without:
+55.0 pts
Closed-book vs. with memory — reader held constant on both sides
Closed-book, the model answers 13.9% of these questions. With Velixar, 68.9%. Holding the reader constant is what makes the difference attributable to memory rather than to model choice.
One reader on both sides. Removing only the memory drops the same model from 68.9% to 13.9% — a +55.0-point lift that belongs to the memory layer. LoCoMo, paired, k=25.
A memory layer that can't show its lift against the same model answering unaided hasn't
demonstrated it's doing anything.
The part nobody expected
The judge — the model that grades the answers — moves scores too: 6.8 points on
identical answers, depending only on who grades.
And the two effects compound in an uncomfortable way. In our grid, Llama-3.3-70B was
simultaneously the weakest reader (58.3%) and the most lenient judge (79.2%
average grade). Weak models grade generously. So the configuration most likely to
flatter a benchmark result is also the cheapest one to run.
Meanwhile the failure everyone names first — a model favouring its own answers — turned
out to be the smallest effect we measured: +0.9 points on average, and two models
were actually harder on themselves. Self-preference is real. It's also roughly twenty
times smaller than reader choice.
The score moves most when you change the reader (22.2 pts), less with the judge (6.8 pts), and least from a model preferring its own output (0.9 pts). The memory layer is held fixed throughout.
Four questions to ask any memory vendor
Including us:
What's your retrieval budget (k)? Recall on this corpus is 89.6% at k=10 and
99.9% at k=30. A score without a budget can't distinguish retrieving well from
retrieving a lot — and k is a cost dial: on this corpus k=25 is roughly 13,800
tokens per query, k=10 roughly 5,500.
Who was the reader? 22.2 points ride on the answer — and reader strength isn't
predictable from release date or tier. In our grid, a newer mini model read worse
than its predecessor.
Who was the judge? 6.8 more points.
What's the lift over the same model closed-book? That's the number that measures
whether the memory layer is doing anything at all.
Recall against its price. k is a cost dial: k=10 buys 89.6% recall at ~5,500 tokens/query; k=25 reaches 98.3% at ~13,800. Only these two budgets were cost-measured, so only these two are plotted — nothing interpolated.
We publish all four for every figure we report. The full methodology, the complete 7×7
grid, the recall curves, and the raw per-cell data are in the two papers this post
summarizes:
The papers behind this post
Retrieved, Not Read — retrieval measured in
isolation, and why we report it separately.
We don't claim a ranking against any other memory system here. The only honest way to
produce one is to run both systems through one harness with one reader and one judge —
and until someone does that work, treat every cross-vendor comparison, including any
that flatter us, as unspecified.