← Lab notes · 2026-07-05 · aLca, Mazemaker
station 06 / the arena
The arena we did not build.
Every number we have published so far came from benchmarks we built, audited, and pinned ourselves — because the published ones kept failing their own audits. That leaves one fair objection standing: of course you win on your own bench. So we walked into someone else’s arena. EverMemBench — their harness, their corpus, their judge, their protocol, unmodified. We wrote a thin adapter into the same slot Mem0, Zep, MemOS and EverMemOS occupy, and ran all 626 questions. This post is the numbers, the caveats stated plainly, and — because this is how we work — every bug the arena surfaced in our engine, each one fixed and shipped to customers before we wrote a word of marketing about it.
Update, same day: we found our own handbrake.
2026-07-05, hours after publishing. The numbers below were real, reproducible — and produced with our strongest rerank channel accidentally disabled. ColBERT blob writes are gated behind an env flag (MM_COLBERT_ENABLED) whose default was left at 0 from a debugging session and forgotten. The recall side happily enables the channel; memories without cached token blobs contribute zero to it, silently, by design. Every number we published this morning — and, embarrassingly and honestly, every recall on our own production pod — ran without the channel our own ablations rank as the single most load-bearing reranker. We found it because the operator refused to believe the bench was this hard, audited the wiring end to end, and checked what the database actually contained: colbert_tokens populated: 0 of 10,222.
With the blobs backfilled and the channel actually firing, same harness, same judge, same protocol:
| Run | Answerer | Category mean | Pooled |
|---|---|---|---|
| As first published (ColBERT silently off) | gpt-4.1-mini | 46.4% | 48.6% |
| Corrected (channel live) | gpt-4.1-mini | 48.7% | 51.0% (319/626) |
| As first published (ColBERT silently off) | deepseek-v4-flash | 51.1% | 52.6% |
| Corrected (channel live) | deepseek-v4-flash | 51.5% | 53.5% (335/626) |
Leaderboard protocol, corrected: 48.7 vs EverMemOS’s 44.6 — four points, on their bench. The fix is not bench-side: the backfill tool now populates any imported corpus, the flag is being promoted to a proper default, and our own 212k-memory production pod got the same backfill the same evening — 211,465 blobs, 2.5 hours, one pg_dump rollback anchor. We publish handbrakes for the same reason we publish wins: if your numbers cannot survive your own audit, they are decoration. Ours moved up.
I. The setup, so you can replicate it
EverMemBench feeds a year of multi-person group-chat dialogue — 10,222 messages across 254 days and multiple rooms — into the memory system under test, then asks 626 questions: 389 multiple-choice, scored by exact option match, and 237 open-ended, scored by an LLM judge (gpt-4.1-nano, per the paper protocol). Answers are generated by gpt-4.1-mini for every system on the leaderboard — the answering model is held constant so the memory system is the variable. The memory system’s only lever is retrieval: which memories land in the context for each question.
We ran the full production engine — C++ ops, GPU BGE-M3 embeddings, ColBERT and DAE rerank channels, Postgres + pgvector — behind their eval/cli.py, retrieval depth k=30. No prompt changes, no scoring changes, no cherry-picking of question subsets. Their harness saves every retrieval, every answer, and every judgment to JSON; ours are on disk and reproducible for around $0.40 per full evaluation.
II. The numbers
| System | Answerer | Category mean | Pooled accuracy |
|---|---|---|---|
| Mazemaker | gpt-4.1-mini (leaderboard protocol) | 46.4% | 48.6% (304/626) |
| EverMemOS (their own system, published) | gpt-4.1-mini | 44.6% | — |
| MemOS (published) | gpt-4.1-mini | 41.6% | — |
| Mazemaker | deepseek-v4-flash (budget) | 51.1% | 52.6% (329/626) |
The apples-to-apples claim first. On the leaderboard protocol — identical answerer, identical judge, their harness — Mazemaker’s category mean is 46.4% against EverMemOS’s published 44.6%, on the benchmark they designed and named after themselves. Our mean is computed over the ten question categories encoded in their own question IDs; their published table groups subtasks slightly differently, so treat the comparison as what it is: same protocol, same models, first place by roughly two points on their home turf.
The budget claim second, framed honestly. Swap the answering model from gpt-4.1-mini ($0.40/$1.60 per million tokens) to deepseek-v4-flash ($0.09/$0.18 — four to nine times cheaper) on the identical retrieval, identical judge, and the score rises to 51.1% mean / 52.6% pooled. This is not leaderboard-comparable — the other systems were not re-run with flash — and we will not pretend it is. What it demonstrates is the system ceiling: the retrieval is good enough that a budget hybrid-reasoner extracts four more points from the same memories. The category that moved most is the one that matters most for a memory product: update tracking — “which value is current after three revisions” — jumped to 50%, fourteen points above anything a non-reasoning answerer managed on any configuration we tested.
III. What the arena broke, and what we shipped
A third-party benchmark is a fuzzer with a leaderboard attached. Running it end-to-end surfaced five real defects in our engine and deployment chain. In keeping with house style — receipts over adjectives — here is the honest list. Every one of these fixes is in 1.0.0-rc7, live on the registry, pulled by every fresh curl | bash install:
Multi-word entities crashed full-text recall. Our FTS channel built a raw tsquery by OR-joining entity strings. Enterprise vocabulary like “Alarm Center” produced invalid syntax, and 145 of 626 queries crashed before we ever scored a point. Single-token benchmarks never tripped it; a year of corporate group chat tripped it in the first hour. Fixed with phrase-token composition.
Idle connections killed the first call after every pause. Postgres’s idle_session_timeout was quietly reaping pooled connections; our pool handed out the corpse on checkout, and the first operation after any ten-minute idle window failed while every health check showed green. The pool now validates connections on checkout. If you have ever seen a “works on retry” bug in a long-lived service, it was probably this shape.
Imported history had no event time. Our read side always supported temporal queries — at_time recall, temporal ranking, date-range filters — but no write path could set a memory’s event time. Every bulk import stamped ingestion time, so a year of chat history landed on one timestamp and the entire temporal machinery ran blind. The write API now accepts per-row event time; imports index when things happened, not when you ran the importer. This is the difference between a memory system and a log.
Dream’s supersedes phase was a silent no-op on Postgres. pgvector returns raw strings when the vector type is not registered on a connection; the parse failure was swallowed, and the phase that detects “this fact replaces that one” checked exactly zero pairs on every Postgres deployment. It now checks hundreds per cycle.
REM flooded cold-start graphs. On a freshly imported corpus with no edges, every memory looks like an orphan and the bridge-discovery phase accepted nearly everything — thirty thousand weak edges in three cycles, which then dragged the DAE embedding channel sideways for the whole corpus. REM now ranks candidates by similarity and budgets each cycle. Live corpora never hit the cap; a bulk import no longer detonates the first night’s sleep.
IV. What dream consolidation actually does, measured
With the fixes in, we ran the clean experiment nobody runs: identical retrieval, identical models, the only variable being whether the corpus had been through dream consolidation. The answer is not a number, it is a shape. Dream helps exactly the categories it was designed for — thematic highlights +9 points, time-point questions +6.7, title recall +4.1 — and costs precision categories, because consolidated summaries crowd specific raw rows out of a fixed top-K. Net on this benchmark: −4 points with dream on everything.
The product conclusion writes itself, and it is now on the roadmap with data behind it: consolidation is not an on/off switch, it is an intent-conditional channel. A “what did we decide about the launch theme” question should see dream-derived memories; a “what is the current budget figure” question should see timestamped raw rows. The engine already classifies query intent. Wiring derived-memory weighting to it is the next lever, and unlike most roadmap items, we know its ceiling in advance: the best-of-all-variants oracle across our runs sits four points above the champion.
V. Two findings for everyone running local models
Retrieval depth is answerer-class-dependent. k=30 gives gpt-4.1-mini +1.9 points over k=10. The same k=30 costs a local qwen2.5:3b answerer 2.1 points — its open-ended accuracy collapses from 32.5% to 24.9% as thirty memories bury it. At matched k=10, the same engine improvements that lifted the frontier-model number also lifted the fully-local, zero-dollar stack (qwen2.5:3b answering, minicpm5 judging): 40.6% versus the 39.9% baseline before the fixes. If you ship local answerers, tune k per model class. We now do.
Hybrid reasoners fail silently on tight token budgets. Our first flash run returned 111 of 626 answers as empty strings — the model spent the entire 1,000-token completion budget reasoning and had nothing left to say, concentrated exactly on the hard multi-hop and temporal questions. The score still read 48.4%, which is the terrifying part: a silently degraded run looks like a plausible result. Give hybrid reasoners at least a 4k completion budget, and grep your answer files for empty markers before you believe any number. This is the same defect class as the reasoning-budget truncation we documented on gpt-5-nano in the Inception Bench post; it is apparently a genre.
VI. Where this leaves the maze
One number from our own bench — LongMemEval-oracle R@5 0.8426 — and now one number from theirs: 46.4 against 44.6 on the leaderboard protocol, 51.1 with a budget answerer, on a benchmark named after the system we beat. Every artifact is a JSON on disk; every engine fix is in a shipped release, not a bench-only patch. The remaining known weakness is multi-hop synthesis (10–20% across every configuration — it needs query decomposition, not a bigger answerer), and the next lever is intent-conditional consolidation with a measured four-point ceiling. The arena did what arenas are for: it found the places we were soft, we hardened them in public, and the number that came out the other side is ours on their turf.