The Jury Nobody Names

The number for the layer

A memory system has one job at retrieval time: when a question is asked, return the evidence that answers it. That is measurable directly, without a language model in the loop, because the benchmark labels which session holds the answer.

98.3%

What k is, and why a score without one is unspecified

k is the retrieval budget: the number of stored items the memory layer is allowed to hand back for a single question. Set k=25 and the system returns its 25 best-matching sessions; the reader sees those and nothing else.

recall@k asks a yes/no question of that budget: of the sessions that actually contain the answer, how many came back inside the top k? It is scored against the benchmark's own evidence labels, so no model is involved in deciding whether the retrieval succeeded.

k is a dial with a real cost on the other side. A larger budget is more likely to include the evidence, and also sends more text to the reader — more tokens, more money, more for the model to sift. A smaller budget is cheaper and risks leaving the answer behind. Where you set it changes the score: on this corpus, recall runs from 89.6% at k=10 to 99.9% at k=30 — nine points of accuracy sit between k=10 and k=25 alone. The full curve is published in Retrieved, Not Read.

So an accuracy figure reported without its retrieval budget is under-specified — you cannot tell whether a system did well because it retrieves precisely or because it was permitted to retrieve a lot. We state k on every number in this paper for that reason, and we would ask the same of anyone else's.

Scoring only questions where every required evidence session came back — many need several — the figure is 97.4% at k=25.

That number needs no reader, no judge, and no prompt. It cannot drift when a model is deprecated. Almost nothing else in this field has that property, and the rest of this paper is about why that matters more than it sounds.

What the field publishes instead

Memory systems are scored end-to-end: feed a long history, ask a question, check the answer. In Measuring Memory we showed that score is a joint measurement of the memory system and the model reading its output, and that changing only the reader moved it 16.4 points. In Retrieved, Not Read we showed the gap between what our retrieval returns and what gets answered is reader-side.

Both of those varied one thing. This time we varied both, exhaustively.

Setup. LongMemEval, 120 questions, k=25. Seven models each generate a full set of answers from identical retrieved context. Every answer set is then graded by every model in turn — 49 cells. Answers are generated once and reused, so the entire grid costs about 3% more than producing the answer sets alone. The diagonal (a model grading its own output) is included deliberately: it is the only way to measure self-preference.

The jury

Row means — each reader's score averaged across judges, so grader noise cancels:

Reader	Score
gpt-5.4	80.6%
DeepSeek-V3.2	79.0%
gpt-5-mini	78.8%
grok-4.3	77.8%
Mistral-Large-3	74.9%
gpt-5.4-mini	71.2%
Llama-3.3-70B	58.3%
Spread	22.2 pts

Same memory. Same retrieved context. Same questions. A 22.2-point range determined entirely by which model reads.

That is larger than the published gap between most competing memory systems. Two systems reporting numbers 15 points apart could be identical underneath, or reversed — the comparison carries no information unless both name their reader, and the field does not consistently do so.

One result worth isolating: gpt-5.4-mini reads worse than gpt-5-mini (71.2% vs 78.8%) despite being the newer model. We reported this once before on a single pair; it reproduces across the grid. Reader strength is not predictable from release date or tier, which means it has to be measured rather than assumed — including by anyone choosing a model to sit downstream of a memory system.

What the judge sees

The grader is the second hidden variable, and it is disclosed even less often than the reader. Column means — each judge's verdict averaged across readers, on identical answers:

Judge	Grades at
Llama-3.3-70B	79.2%
gpt-5-mini	75.7%
Mistral-Large-3	74.7%
gpt-5.4	73.6%
gpt-5.4-mini	72.5%
DeepSeek-V3.2	72.5%
grok-4.3	72.4%
Spread	6.8 pts

The same answers score 6.8 points apart depending only on who grades them. Smaller than the reader axis — and still wide enough to reorder published results.

Self-preference: the smallest of the three

The obvious worry with LLM-as-judge is a model favouring its own output. We measured it directly — each model's diagonal cell against its own off-diagonal row mean:

Model	Self-preference
Mistral-Large-3	+3.5 pts
Llama-3.3-70B	+1.7 pts
gpt-5-mini	+1.2 pts
gpt-5.4-mini	+1.2 pts
DeepSeek-V3.2	+0.1 pts
gpt-5.4	−0.6 pts
grok-4.3	−1.1 pts
Mean	+0.9 pts

It is real, it is not universal — two models are harder on themselves — and it is roughly twenty times smaller than reader choice. We report it plainly because we had assumed the opposite: self-preference is the failure people name first, and it is the least of the three effects. A separate judge is still correct practice. It is not where the uncertainty lives.

What this means for retrieval systems

Put the three numbers together — 22.2 points of reader, 6.8 of judge, 0.9 of self-preference — against a retrieval layer measured at 98.3%, and a structural problem appears.

When retrieval approaches saturation, further retrieval improvement becomes invisible end-to-end. If the evidence is already being returned, a better retriever has nothing left to add to the answer; the remaining error is downstream. We observed this directly: on LongMemEval's multi-session questions — the most retrieval-dependent category — swapping gpt-5-mini for the stronger gpt-5.4 moved the score by exactly zero (71.9% both). Where retrieval is the binding constraint, reader strength buys nothing. The converse is what should worry the field: where the reader is the binding constraint, retrieval quality buys nothing either, and an end-to-end benchmark cannot tell you which regime you are in.

So the industry convention actively hides retrieval progress. A system that improves recall from 90% to 98% may show no end-to-end movement at all, and a system that improves nothing may post a large gain by changing readers.

Two consequences we act on:

Report the layer you built. Retrieval recall at a stated k measures our layer. End-to-end measures our layer times someone else's model. Both are legitimate; only one of them is about us.

Retrieval budget is a cost dial, not just an accuracy dial. Recall is 89.6% at k=10 and 99.9% at k=30. Context in this corpus runs about 553 tokens per session, so k=25 costs roughly 13,800 tokens per query and k=10 roughly 5,500. Reporting accuracy without the budget hides whether a system is accurate because it retrieves well or because it retrieves a lot.

What we claim

We claim that on LoCoMo our retrieval returns the labelled evidence for 98.3% of questions at k=25, and the complete evidence set for 97.4%. We claim that holding the reader constant, memory is worth +55.0 points against the same model answering closed-book. We claim the reader and judge effects reported above, measured on 120 questions across all 49 cells of a complete 7x7 grid.

We do not claim that 98.3% is comparable to any published end-to-end score — they are different quantities. We do not claim a ranking against any other memory system on these benchmarks; the only honest way to produce one is to run both systems through one harness with one reader and one judge, which is work we have not done. We do not claim these reader or judge rankings generalize beyond this task shape — the same models reorder on different work.

Reproducibility

Every figure came through the standard client path against the shipping product. Retrieval budget, sample size, reader, and judge are stated because a benchmark result that omits them cannot be checked. The full 49-cell matrix and per-question recall data are published alongside this paper as raw JSON.

Three measurement defects in our own harness were found and corrected while producing this grid, including one where reading result files while they were still being written produced a plausible-looking table over a silently reduced denominator. It is the same failure in every case — scoring against a denominator you have not verified — and it is the one we now gate for explicitly. We mention it because a paper arguing that hidden variables corrupt published numbers should say which of its own it had to go looking for.

@techreport{velixar-the-jury-nobody-names-2026,
  title       = {The Jury Nobody Names},
  author      = {Velixar AI},
  year        = {2026},
  institution = {Velixar},
  url         = {https://velixarai.com/research/the-jury-nobody-names}
}