We Thought a Bigger AI Model Would Win. The Benchmark Said Otherwise.
We benchmarked Velixar on HaluMem expecting a bigger extraction model to win. It didn't — and our obvious fix for the weakest category made things worse. What component isolation taught us.
Published July 2026 · Engineering
When people talk about AI memory, they usually focus on one question:
Can the system remember?
That's an important question — but it's not the interesting one.
The real questions are:
- Does it remember the right things?
- Can it ignore information that shouldn't be remembered?
- Can it recognize when knowledge changes over time?
- And perhaps most importantly, how much of its performance actually comes from the memory system versus the language model attached to it?
Those were the questions we set out to answer when we benchmarked Velixar against the current generation of AI memory systems using the HaluMem benchmark.
Some of the results were exactly what we hoped.
Others challenged assumptions we'd been carrying for months.
And one of our most obvious ideas for improving performance actually made the system worse.
Why We Chose HaluMem
Most AI benchmarks measure language models.
HaluMem measures something different.
Instead of asking whether an LLM can answer questions, it evaluates whether a memory system can successfully capture information, retrieve it later, update outdated knowledge, and avoid storing false information in the first place.
In other words, it measures whether an AI memory layer behaves like memory instead of acting as an oversized prompt.
That's exactly the problem Velixar was built to solve.
The Headline
Our benchmark produced two very different stories.
Velixar achieved the highest extraction performance on the published Halu-Medium leaderboard. Our end-to-end memory performance — question answering and memory updates — landed squarely in the middle of the pack.
| Category | Velixar | Best published | Standing |
|---|---|---|---|
| Extraction F1 ↑ | 90.7% | MemOS 79.7% | Best published |
| Question Answering ↑ | 53.1% | MemOS 67.2% | Mid-pack |
| Memory Updates ↑ | 33.4% | MemOS 62.1% | Mid-pack |
| False-Memory Resistance ↑ | 42.8% | Memobase 80.8% | Below field |
Scope: three-user stratified pilot on Halu-Medium, not the full user set. Published leaderboard figures were scored with gpt-4o; our panel uses gpt-5-family models, so this is not a byte-for-byte reproduction of the paper. The two invariance tests below are our argument that the substitution doesn't drive the result.
At first glance, those results seem contradictory.
How can a system be exceptionally good at capturing information yet only average when using that memory later?
That question ended up driving the rest of our investigation.
First, We Asked Whether the Benchmark Was Fooling Us
Whenever benchmark results look unusually good, there's an obvious concern.
Maybe the evaluator is simply being generous.
Modern AI benchmarks increasingly rely on language models as judges, and different judges can produce different scores.
So we tested exactly that.
Instead of relying on a single evaluator, we ran multiple independent judges and measured agreement between them.
The result surprised us.
The judges agreed remarkably closely — our three-model panel reached Fleiss' κ = 0.80, rising to 0.93 with a fourth judge added.
Even more interesting, the strongest evaluation model actually scored memory extraction slightly higher, not lower.
There wasn't any evidence that stronger models were artificially inflating Velixar's results.
The extraction numbers remained essentially unchanged regardless of who evaluated them.
That gave us confidence that the benchmark wasn't measuring evaluator bias.
It was measuring the system itself.
Then We Asked the Bigger Question
Was Velixar actually responsible for those extraction numbers?
Or were we simply benefiting from using a stronger extraction model?
This is a surprisingly common blind spot in AI benchmarking.
Many published results report the final score without separating the contribution of the memory architecture from the contribution of the language model performing extraction.
We wanted to know exactly where the performance was coming from.
So we swapped the extractor.
Everything else stayed the same.
The Bigger Model Barely Helped
Our expectation was straightforward.
A larger, more capable language model should produce substantially better memories.
Instead, we discovered something unexpected.
The larger model extracted 67% more memories — 8,981 against 5,383.
That sounds impressive.
Except those additional memories produced almost no measurable improvement in quality.
Extraction accuracy was identical at 96.3%. Precision moved two-tenths of a point in the wrong direction. Overall F1 changed by one-tenth of a point.
Even more surprising, downstream performance became slightly worse. Those additional memories introduced retrieval noise that cost roughly a point on both update accuracy and question answering.
The takeaway was surprisingly simple.
More memories do not automatically produce better memory.
The architecture mattered far more than the size of the extraction model.
That may have been the single most important result in the entire benchmark — and it's the one we'd encourage other teams to reproduce on their own systems, because it's cheap to run and it changes how you read every extraction number in the field, including ours.
The Fix That Made Everything Worse
Our weakest benchmark category was memory updates.
The obvious solution seemed almost trivial.
When new information replaces old information, simply delete the outdated memory.
Problem solved.
Or so we thought.
We ran exactly that as a controlled ceiling test.
Performance dropped. Memory updates became less accurate by 7 points. Question answering dropped by 3.
Why?
Because deleting outdated information also removed valuable context.
The system hallucinated less — but it omitted more.
The experiment taught us something important.
The difficult problem isn't deleting obsolete memories.
The difficult problem is determining exactly which memory has truly become obsolete while preserving everything else that still provides useful context.
Temporal reasoning turned out to be substantially more complicated than simple replacement — and this is the result we'd most want other teams to take seriously, because it's the fix almost everyone reaches for first.
Was the Benchmark Measuring Us, or Missing Us?
At that point we stopped looking at benchmark scores and started interrogating the result itself.
One question kept bothering us.
Were these results exposing a real limitation in Velixar?
Or was the benchmark simply failing to exercise capabilities we'd already built?
We checked. It was the former.
Our temporal handling is genuinely incomplete, and the update and supersession results reflect real system behavior rather than an evaluation artifact.
That distinction matters more than it might appear. "The system has a limitation" and "the harness didn't exercise the feature" produce identical-looking scores and mean entirely different things. A benchmark result that leaves those two possibilities open isn't a result — it's an ambiguity.
We'd rather resolve it in public and say plainly which one we're looking at.
Temporal reasoning is on our roadmap. It was already on our roadmap. What the benchmark changed is our understanding of what the hard part actually is.
What We Learned About False Memories
Another discovery involved false-memory resistance.
Initially we wondered whether stronger language models might naturally become better at filtering misinformation.
The data didn't support that idea.
False-memory resistance came out effectively identical — to the decimal — regardless of which extraction model we used.
That tells us something important.
The limitation isn't coming from the language model. It's a property of where a system chooses to sit on a trade-off that every memory system faces.
Look at the leaderboard and the shape is unmistakable. The systems that beat us on false-memory resistance do it by capturing dramatically less. The top scorer on that metric reaches 80.8% resistance while extracting 14.6% of available facts — barely a seventh of what we capture. Every point of resistance on that board was bought with recall.
That's a real trade-off, not a defect, and we've deliberately sat at the high-capture end of it. But it's a position that only holds up if the capture is paired with judgment about what's worth keeping, and improving that balance is active engineering work for us.
Not a model problem.
A systems problem.
The Bigger Lesson
This benchmark ended up teaching us something broader than where Velixar sits on a leaderboard.
Modern AI memory systems are really three systems working together:
- The model that extracts information.
- The memory architecture that stores and organizes it.
- The model that retrieves and reasons over it.
If you change all three at once, you don't actually know what's improving.
You only know that the final score changed.
Throughout this benchmark we intentionally isolated each variable.
We held the judges constant while changing the extractor.
We held the extractor constant while changing the judges.
We tested architectural changes independently instead of assuming they would help.
Some ideas worked. Some didn't.
The important part is that we now know why.
Where Velixar Stands Today
We're proud that Velixar delivers best-in-class extraction performance on the Halu-Medium benchmark.
Just as importantly, we've verified that those results aren't dependent on expensive frontier models or unusually favorable evaluators. A number that doesn't move when you swap the model underneath it is a property of the system. That's the only kind of number worth putting on a scorecard.
At the same time, we're equally clear about where we have work to do.
Temporal reasoning. Memory updates. False-memory resistance.
Those aren't mysteries anymore. We know where the gaps are, and we know they're architectural challenges — not problems that disappear by swapping in a larger language model.
For us, that's the real value of benchmarking.
Not collecting impressive numbers.
Understanding exactly what to build next.
Engineering Over Headlines
It's tempting to publish only the best number on a leaderboard.
Many companies would.
We chose not to.
The most valuable outcome from this benchmark wasn't discovering where Velixar leads.
It was discovering where our assumptions were wrong.
We assumed bigger models would produce dramatically better memories. They didn't.
We assumed naïve temporal supersession would improve updates. It made them worse.
We assumed evaluator choice might explain our extraction lead. It didn't.
Those findings are now shaping the next generation of Velixar's memory architecture.
That's ultimately what engineering benchmarks should accomplish.
Not proving you're finished.
Showing you exactly where to go next.
The Short Version
We put Velixar through HaluMem, a test built to measure memory systems rather than the AI models attached to them — can it capture what matters, find it again later, notice when something has changed, and refuse to store things that were never true. Velixar came out best-in-class at capturing information, ahead of every published system on the board. On the harder job of using that memory afterward — answering questions and updating what it already knows — we landed in the middle of the pack. Before publishing either result we checked the two things that most often make benchmark numbers misleading: whether a generous grader was inflating our score, and whether we were simply borrowing performance from an expensive AI model. Neither was true. The result holds up regardless of who grades it or which model does the work, which is what makes it a fact about our system rather than about the technology we happened to run it on.
Two findings changed how we think. Paying for a much larger AI model produced 67% more memories and no measurable improvement in quality — and slightly worse answers, because the extra material got in the way. And our most obvious fix for the weakest score, deleting outdated information when something replaces it, made performance drop rather than rise: the system made fewer things up, but left more out, because deleting the old fact took useful context with it. Both point in the same direction. The hard problem in AI memory isn't storing more or storing faster. It's judgment — knowing which pieces matter, which have quietly stopped being true, and which still deserve to be kept. That's the part we're building next, and it isn't something a bigger model will solve for us.