The Corpus Outlives the Model

The asset that survives the swap

Every organization running agents in production has now lived through at least one model migration. A better model ships, the cost curve moves, a provider changes terms, a region loses a deployment — and the reader gets swapped. What does not get swapped is the corpus. The accumulated record of what a system has learned about a customer, a codebase, an aircraft fleet, a student, is the part of the stack that persists across every one of those migrations.

Inference is commoditizing. That is not a prediction we need to defend; it is visible in the price sheets. And as it commoditizes, the durable value in an agent system migrates toward the layer that holds state — the memory and knowledge layer, whose worth is precisely its ability to survive the model underneath it changing.

Which creates a measurement problem the field has not solved. Almost every memory benchmark number published today is a single end-to-end accuracy figure: retrieve context, hand it to a language model, let another language model grade the answer, report one percentage. That number is a joint measurement of at least three systems — the memory layer, the reader, and the judge — and the field routinely names only the first, and often not even that.

If the reader is the part that changes and the corpus is the part that lasts, then an evaluation that cannot separate the two cannot tell you what you own.

What the memory layer does on its own

Start with the measurement that has no model in it at all.

On LoCoMo, 310 questions, retrieval budget k=25:

• Recall@25: 98.3% — the supporting evidence for the question is present in the retrieved context.
• Complete evidence set: 97.4% — every piece of evidence the question requires is present, together, in the same retrieved context.

The second number is the one that matters and the one almost nobody reports. Multi-evidence questions are where retrieval systems usually come apart: a system can look excellent on "did we surface something relevant" while routinely delivering three of the four facts an answer actually needs. The distance between those two conditions is where a memory layer either holds or doesn't. Ours costs 0.9 points.

Neither figure involves a reader or a judge. Both are computed against the retrieved context itself, so they are properties of the memory layer and nothing else — they do not move when you change models, which is precisely what makes them the right instrument for an asset defined by surviving model changes.

And both are stated at a budget, because k is a cost dial rather than a constant. On the same run, recall moves 89.6% at k=10, 96.5% at k=20, 98.3% at k=25, 99.9% at k=30. Nine points sit between k=10 and k=25. A retrieval score published without its budget is under-specified by roughly the width of the entire competitive field.

What happens when a model reads it

Now put a model in the loop. Same corpus, same retrieval, same questions: end-to-end answer accuracy on LoCoMo is 68.9%.

Retrieval places the complete evidence set in front of the reader 97.4% of the time. The reader converts that into an answer graded correct 68.9% of the time. Those are not the same quantity and the gap is not a subtraction — recall asks whether the evidence arrived, accuracy asks whether a language model then used it and a second language model agreed. But the shape is unmistakable: the evidence is almost always there, and roughly three in ten times something downstream of our layer fails to turn it into an answer.

That downstream stage is a component we do not sell, do not control, and the customer will swap. Which raises the only question that matters for valuation: how much does that conversion stage vary?

The same-reader control gives the first half of the answer. Holding the reader and everything else fixed and removing only the memory, the same questions score 13.9% — a +55.0 point lift attributable to the memory layer, paired, same model both arms. No competitor publishes this figure, because running a no-memory control arm requires deliberately measuring your own system's absence.

The 7×7 grid gives the second half.

What we ran

We built a 7×7 reader × judge matrix on LongMemEval — 120 questions, retrieval budget k=25, every reader's answer set graded by every judge. Forty-nine cells. The panel is seven models across five vendors: gpt-5.4, gpt-5-mini and gpt-5.4-mini (OpenAI), DeepSeek, Llama-3.3-70B (Meta), Mistral, and Grok (xAI). Answer sets are generated once (~1.9M tokens each) and then judged by all seven graders, so the full grid costs roughly three percent more than a single run's generation. The diagonal is included deliberately: reader-equals-judge is the only configuration that measures a model's preference for its own output.

The grid was not an academic exercise. It started as a bug fix. Our own LongMemEval harness had a single model constant — gpt-5-mini — serving both the answering stage and the grading stage, which meant gpt-5-mini was grading gpt-5-mini's work. That was the strongest methodological objection available against a number we had already published. We fixed it, measured the damage directly at −0.8 points, and then decided that a defect we had shipped once was worth generalizing into a measurement rather than a patch note.

Everything now runs through one shared dispatch module, so the reader and the judge are explicit arguments with a guard that refuses to let them silently collapse into the same deployment.

Three axes, and only one of them is us

Reader spread: 22.2 points. Identical memory, identical retrieval, identical questions — only the model reading the retrieved context changes.

Reader	Mean score (across judges)	Self-preference
gpt-5.4	80.6%	−0.6
DeepSeek	79.0%	+0.1
gpt-5-mini	78.8%	+1.2
Grok	77.8%	−1.1
Mistral	74.9%	+3.5
gpt-5.4-mini	71.2%	+1.2
Llama-3.3-70B	58.3%	+1.7

That range is wider than the published gap between most competing memory systems. A memory benchmark score reported without a named reader is not a weak claim — it is an uninterpretable one, because the dominant term in the number is unstated.

It also answers the question the previous section left open. Retrieval delivers the complete evidence set 97.4% of the time regardless of what happens next — that number does not move across this table. The conversion of that evidence into a graded answer moves by 22.2 points depending on which model you hand it to. One of those two figures is a property of the system being sold. The other is a property of a component that will be replaced.

Judge spread: 6.8 points. The same seven answer sets, graded seven ways.

Judge	Mean score awarded
Llama-3.3-70B	79.2%
gpt-5-mini	75.7%
Mistral	74.7%
gpt-5.4	73.6%
gpt-5.4-mini	72.5%
DeepSeek	72.5%
Grok	72.4%

Smaller than the reader axis, still larger than several published system-to-system gaps — and this is the axis nobody discloses at all. Swapping Grok for Llama as your grader is worth nearly seven points on identical work.

Self-preference: 0.9 points, mean. Real, not universal, and the smallest of the three. gpt-5.4 (−0.6) and Grok (−1.1) are harder on themselves than their peers are; Mistral is the worst offender, inflating its own score by 3.5 points. We had called self-preference the most serious objection to our own published figure. The grid says it is worth fixing and worth disclosing, and it is the least important of the three effects. Fix it; don't lead with it.

Margins exclude the diagonal and are computed from unrounded cell values; displayed figures are rounded to one decimal. Per-cell results are in [The Jury Nobody Names].

Two results we did not expect and are keeping in the record because they are inconvenient. Llama-3.3-70B is simultaneously the weakest reader in the panel (58.3%) and the most lenient judge (79.2%) — weak models grade generously, so a weak-reader/weak-judge pairing partially masks its own weakness, which is the exact configuration a vendor optimizing for a headline would drift into unintentionally. And gpt-5.4-mini reads measurably worse than gpt-5-mini — 71.2% against 78.8% — despite being the newer model, reproducing across a full grid a non-monotonicity we had previously observed in a single pair. Model version numbers are not an ordering.

The saturation trap

There is a structural reason end-to-end scores mislead, independent of disclosure.

When retrieval is the binding constraint, reader strength buys nothing. When the reader is the binding constraint, retrieval improvement is invisible. We have both regimes in the same dataset: upgrading the reader from gpt-5-mini to gpt-5.4 moved the multi-session category by exactly zero — 71.9% before, 71.9% after — while knowledge-update, temporal and single-session-assistant all moved several points. The retrieval-bound category did not care which model read the context.

An end-to-end benchmark cannot tell you which regime you are in. It reports one number in both cases. So a memory system that meaningfully improves retrieval can show no end-to-end gain, and a memory system that has not improved at all can show a large one on the back of a reader upgrade. Both errors are currently indistinguishable in published results.

The standard we're proposing

Not a new benchmark. A disclosure and decomposition discipline that can be applied to any of the existing ones.

1. Name the reader. Model and deployment. A score without one carries a 22-point ambiguity.

2. Name the judge, and prove it isn't the reader. Enforce it in code, not in the write-up. Report the self-preference cost if you can measure it.

3. Publish a reader-independent retrieval measure, at a stated budget, under both conditions. Recall@k isolates the memory layer from the model that consumes it — but report the complete-evidence-set rate alongside it, not just the any-evidence rate. Ours are 98.3% and 97.4% at k=25 on LoCoMo. A system whose two figures diverge sharply is telling you it surfaces relevant material without assembling it, and only the stricter number predicts behavior on the multi-hop questions real deployments are made of. State k in both cases.

4. Publish a same-reader, no-memory control. The only quantity attributable to the memory system is the difference the memory makes with everything else held fixed. Ours is +55.0 points — 68.9% with memory against 13.9% closed-book, same reader, paired. A raw end-to-end score is a property of the pair; the lift is a property of the layer.

5. Report the spread, not just the point. The variance across readers and judges is the finding. Collapsing a grid to its best cell is selection, and the field currently has no way to detect it.

6. Don't report what your ingestion can't support. We withdrew our own temporal sub-score: single-pass ingestion collapses the time axis, so the figure was measuring the harness, not the memory system's temporal handling. Measuring real temporal capability requires back-dated ingest or a corpus accumulated over actual elapsed time. We would rather publish the gap than the artifact.

The obvious objection

We benefit from this. A vendor whose retrieval measures 98.3% and 97.4% has an obvious interest in arguing that retrieval should be measured separately, and we should say so before someone says it for us.

Two things keep it honest. First, recall@k and end-to-end answer accuracy are not the same quantity, and we will not place them side by side against anyone else's end-to-end figure — that is the apples-to-oranges error we are objecting to, merely pointed in our favor. Second, the same discipline that flatters our retrieval number is the one that forced us to cut a temporal claim, disclose a self-grading defect in our own harness, and revise a published headline downward as the protocol tightened. A standard you only apply when it helps is not a standard.

The comparison that would actually settle anything is running competing systems through one harness with one named reader and one named judge. That is buildable — several of these systems are open source — and it is where we think this goes next.

Why it matters beyond benchmarks

If model capability is converging and model cost is falling, then the question a buyer should be asking is not "what does your system score." It is "what does your system score, on which reader, graded by whom, at what retrieval budget, against what no-memory baseline — and how much of that number comes back with me when I change models next quarter?"

That last clause is the whole thesis. Adaptability is the property being purchased. A memory layer's value is the portion of its performance that transfers when the inference underneath it is replaced, and no single end-to-end number can express that portion.

Ours is stated in two figures that do not move when the model does — 98.3% recall and a 97.4% complete evidence set at k=25 — and one that quantifies what the memory itself is worth to any reader you put in front of it: +55.0 points against the same model with no memory at all. Everything else in this post is the variance around those numbers, contributed by parts of the stack that will not be the same in a year. Component isolation is not methodological fussiness. It is the only way to price the part that lasts.

---

Retrieval figures: LoCoMo, n=310, k=25 — recall@25 98.3%, complete evidence set 97.4%, end-to-end 68.9%, closed-book control 13.9%. Reader × judge grid: LongMemEval, n=120, k=25. The two datasets measure different things and their numbers are not interchangeable. Grid data, per-cell results, and the recall@k curve are documented in [The Jury Nobody Names] and [Retrieved, Not Read]. Panel: gpt-5.4, gpt-5-mini, gpt-5.4-mini, DeepSeek, Llama-3.3-70B, Mistral, Grok — seven models, five vendors, each serving as both reader and judge. Deployment identifiers, raw cells, and the grid renderer are available on request.

@techreport{velixar-the-corpus-outlives-the-model-2026,
  title       = {The Corpus Outlives the Model},
  author      = {Velixar AI},
  year        = {2026},
  institution = {Velixar},
  url         = {https://velixarai.com/research/the-corpus-outlives-the-model}
}