What the Numbers Actually Say
Velixar on HaluMem — extraction leads the field, QA and Update don't, and we ran down both confounds before publishing either result.
Bottom line
Velixar's extraction is best-in-class, and — the part that matters more — it is robust to extractor choice. A cheap model matches an expensive one to within a tenth of a point. On the metrics that stress the memory system rather than the model in front of it, end-to-end QA and Update, Velixar lands mid-pack. Both weak spots are structural: false-memory resistance and update-omission. Neither is fixed by a bigger model, and the obvious fix for the second one — wiring temporal supersession — makes things worse, not better, when done naïvely. We measured that too.
| Velixar | Best published | Standing | |
|---|---|---|---|
| Extraction F1 ↑ | 90.7% | MemOS 79.7% | Leads field |
| False-memory resistance ↑ | 42.8% | Memobase 80.8% | Weak spot |
| QA Correct ↑ | 53.1% | MemOS 67.2% | Mid-pack |
| Update Correct ↑ | 33.4% | MemOS 62.1% | Mid-pack |
gpt-5-mini extractor · 3-judge consensus · 3-user stratified pilot · compared against the published gpt-4o-based Halu-Medium leaderboard; our panel is gpt-5-family, so this is a standing indication, not a byte-for-byte reproduction.
Four questions, four measured answers
01 — Where does Velixar stand against the leaderboard?
Extraction leads; QA and Update are mid-pack.
Extraction Recall, Precision and F1 beat every published system on Halu-Medium, including the paper's top performer, MemOS. On the end-to-end tasks that actually exercise storage and retrieval, Velixar sits level with Mem0 and Supermemory and behind MemOS and Zep. The extraction headline is real. It is not the whole story, and we are not going to quote it as though it were.
02 — Is the result just an artifact of a lenient judge?
No. Judge choice barely moves it.
Run across a three-model judge panel, per-judge scores cluster tightly: QA agreement is Fleiss' κ = 0.80, rising to 0.93 with a fourth judge added. And the direction rules out the effect people usually worry about — the strongest judge scored recall higher, not lower. There is no "stronger model inflates the score" gradient here. The numbers are robust to who grades them.
This is worth stating plainly because it is the axis nobody in this field discloses at all, and because it cuts against a hedge we had been carrying ourselves. We expected judge strength to be a live confound. On this benchmark it isn't.
03 — Would wiring temporal supersession fix the Update gap?
No. Naïve supersession makes it worse.
The intuitive fix for a 33% Update-Correct score is to delete the stale fact whenever a new one supersedes it. We ran that as a ceiling test — delete at every update point — and it reduced Update-Correct by 7 points and QA-Correct by 3.
It cut hallucination exactly as intended. It raised omission by more. Removing the stale memory took needed context out with it. Supersession is not a free win, and the hard part was never the deletion: it is the precision of deciding which fact is genuinely superseded, and de-ranking only that one.
04 — How much of the extraction win is the LLM rather than Velixar?
Almost none. The extractor barely matters.
Swapping the cheap gpt-5-mini extractor for gpt-5.4 produced 67% more memories and near-identical quality: F1 within 0.1 points, false-memory resistance identical to the decimal. The extraction lead is the atomic-faithful approach plus the Velixar store, not a strong-model cherry-pick.
The extra memories were not merely redundant — they were slightly harmful, costing about a point on Update and QA through retrieval noise.
Standing vs the published leaderboard
Halu-Medium. Velixar = gpt-5-mini extractor, 3-judge consensus. ↑ higher is better.
| System | Recall↑ | Prec↑ | Acc↑ | F1↑ | FMR↑ | Upd-C↑ | QA-C↑ |
|---|---|---|---|---|---|---|---|
| Velixar | 86.5 | 95.4 | 96.4 | 90.7 | 42.8 | 33.4 | 53.1 |
| MemOS | 74.1 | 86.3 | 59.6 | 79.7 | 44.9 | 62.1 | 67.2 |
| Zep | — | — | — | — | — | 47.3 | 55.5 |
| Mem0-Graph | 43.3 | 87.2 | 61.9 | 57.9 | 55.7 | 24.5 | 54.7 |
| Mem0 | 42.9 | 86.3 | 60.9 | 57.3 | 56.8 | 25.5 | 53.0 |
| Supermemory | 41.5 | 90.3 | 60.8 | 56.9 | 51.8 | 16.4 | 54.1 |
| Memobase | 14.6 | 92.2 | 32.3 | 25.1 | 80.8 | 5.2 | 35.3 |
Memobase is the clearest illustration of the trade-off the field rarely draws: the best false-memory resistance on the board, 80.8%, bought by extracting almost nothing — 14.6% recall, 25.1% F1. Our 42.8% is the other end of the same curve.
Scope: 3-user stratified pilot, not the full 20-user set. Published leaderboard figures are gpt-4o-based; our panel uses gpt-5-family models, so this is not a byte-for-byte reproduction of the paper. Questions 02 and 04 are the argument that the substitution does not drive the result.
The extractor comparison
Same pilot, same primary judge (gpt-5-mini), full census on both sides. Δ = gpt-5.4 − gpt-5-mini.
| Metric | gpt-5.4 | gpt-5-mini | Δ |
|---|---|---|---|
| Memories extracted | 8,981 | 5,383 | +67% |
| Extraction Recall | 87.2 | 86.9 | +0.3 |
| Target Precision | 95.2 | 95.4 | −0.2 |
| Accuracy | 96.3 | 96.3 | 0.0 |
| Extraction F1 | 91.0 | 90.9 | +0.1 |
| FMR (interference) | 43.3 | 43.3 | 0.0 |
| Update Correct | 37.3 | 39.2 | −1.9 |
| QA Correct | 52.8 | 53.8 | −1.0 |
A far larger, more expensive extractor delivers statistically identical quality and slightly worse end-to-end results. The win lives in the approach, not the model.
There is a general point buried in that table. The extractor is a swappable model component sitting upstream of the memory layer, exactly as the reader and judge sit downstream — and its contribution to a published extraction score is, in almost every benchmark report in this field, entirely unmeasured. Ours is now measured, and it is 0.1 points. A number that does not move when you replace the model is a property of the system. That is the only kind of number worth putting on a scorecard.
Why the weak spots are real
The point of reading these two numbers carefully is to resolve an ambiguity the field usually leaves open: is a low score a real limitation of the system, or an artifact of a harness that never exercised the relevant capability? For both of Velixar's weak spots, the answer is the former. They reflect genuine, current behavior of the deployed system — not a test that fell short of it.
The Update and temporal result is the clearest case. Velixar's handling of superseding facts is genuinely incomplete today: when a later fact revises an earlier one, the system does not yet reliably retire the stale version, and the mid-pack Update number is an honest reflection of that. We would rather name the gap than let the result sit ambiguous between "the system has a limitation" and "the test didn't reach the feature." Those are different claims, and buyers deserve to know which one they are reading.
False-memory resistance is the same honesty seen from the other side. As the leaderboard table shows, resistance and extraction recall trade against each other: Memobase buys 80.8% resistance by extracting almost nothing — 14.6% recall — while Velixar sits at the high-recall end of that curve. Our 42.8% is not a defect to explain away; it is the operating point a high-recall posture puts you at, and it is model-independent — 43.3% on both extractors, to the decimal. It is a property of where we chose to sit on the curve, and moving along it is a deliberate product decision, not a model upgrade.
Both numbers are ones we can locate and own, and neither is closed by a bigger model. That is the honest version of a benchmark result, and it is the version we would want to read about someone else's system.
What this run demonstrates beyond its own scorecard
Three numbers here were produced by holding a model constant and swapping another: judge invariance (κ = 0.80), extractor invariance (ΔF1 = 0.1), and the supersession ceiling test that returned a negative result we published anyway.
That is the whole method. A memory system's score is a joint measurement of the memory, the model that extracts into it, and the models that read from and grade it. Report the joint number alone and you cannot tell a system improvement from a model upgrade — in either direction. Pin the components one at a time and the number that survives is the one you actually own.
On this benchmark, what survived is a best-in-class extraction result that holds across judge panel and extractor tier, and a mid-pack end-to-end result with two limitations we can name and own. Both belong in the same post.
Benchmark: MemTensor/HaluMem, Halu-Medium (IAAR-Shanghai/HaluMem-Medium). Scope: 3-user stratified pilot. Leaderboard standing scored by 3-judge consensus; the extractor comparison uses the primary judge at full census on both sides, the two bases differing by no more than 0.2 points on shared metrics. Full per-metric results available on request.