I build with AI coding agents. I do not write much syntax by hand. That is relevant to this story, because the agent I was using to build a memory system also wrote its evaluation, and I believed it.
The system is called NSM. It is a memory for AI agents that lives in one SQLite file on your own machine: no vector database, no API key, no bill. The pitch was that a graph of concepts with "spreading activation" could replace embeddings. The agent that built it, Gemini inside Antigravity, produced a README with a benchmark table: NSM 91% on LoCoMo, Mem0 15%, Zep 26%. Sub-2 ms latency. "100% non-LLM, biologically inspired."
I posted about it. Developers told me I was a rookie for trusting it. They were rude, and they were right about the table.
What the table actually was
When I had a different agent audit the code, this is what it found in the benchmark script:
} else {
recordScore("Mem0 Cloud (Live SDK)", 15.0, 0, 354.8, scenario.category);
}If there was no Mem0 API key, the script recorded 15.0 for Mem0. The README published that as a "Live SDK" result. The "LoCoMo" score was ten hand-written scenarios graded by keyword matching, not the LoCoMo dataset. A "pheromone strength" chart climbed to 4.88 in code that caps the value at 1.0.
None of this was hidden. It was in plain TypeScript in the repo the whole time. I just had not read it, because the table said what I wanted it to say.
Measuring it properly
I already had a harness I trust, built for MemVault, my hosted memory product. It runs any memory system on the public LoCoMo dataset (885 held-out questions over long multi-session conversations), with a fixed answer model, a separate judge, and the answer and judge prompts copied verbatim from Mem0's own published evaluation code. Every system gets the same questions, the same answerer, the same grader. Comparisons are paired: for each question, did A get it right and B wrong, or the reverse, and is the imbalance bigger than chance (exact McNemar test).
I deleted the fake evaluation, plugged NSM into that harness, and ran it. First result, exactly as built:
| NSM as built (test split) | 5 memories | 10 | 20 | 50 |
|---|---|---|---|---|
| Accuracy | 60.5% | 68.1% | 75.0% | 80.3% |
Not 91%. But also not nothing: at every size it beat plain embedding search run through the same harness (52.0% / 60.6% / 68.5% / 77.1%), with zero model calls, in 27 ms. The thing was real. The table was not.
Then I turned the graph off. The "biologically inspired" spreading activation, the part the whole pitch rested on:
| 5 | 10 | 20 | 50 | |
|---|---|---|---|---|
| Graph on | 60.5 | 68.1 | 75.0 | 80.3 |
| Graph off | 61.0 | 67.9 | 75.8 | 80.8 |
No difference at any size (p = 0.30 to 0.83). BM25 and a substring match were doing all the work. The ant colony was decoration.
Twenty-seven experiments
So I did what I should have done from the start: one change at a time, each one its own run on a development split, each one paired against the baseline, nothing promoted to a default until the test split confirmed it. About $45 of API calls for the answer model and judge over two days.
What did not help, and is now documented as not helping:
- Weighting the memory's first line higher in BM25. Zero effect. It turned out the weights had been applied to the wrong column for the entire life of the project. Nobody could have noticed without a paired run that came back with zero discordant questions.
- Episode expansion (adding the turns around each hit). Hurt by 4 to 5 points at small sizes, p < 0.001, because the neighbours pushed real hits below the cutoff.
- A second graph design with one node per distinctive word and ACT-R style fan damping so hub words could not flood the mesh. Nothing. A third variant that let the graph add memories rather than just reorder them made things slightly worse.
- A stronger reranker (bge-reranker-base). Three to five seconds per query on a CPU, and worse at 20 memories than the small one. Rejected.
What did help:
- Porter stemming in the index: +2 to +3 points at small sizes on dev (p = 0.003 at 20 memories).
- Keeping concept nodes out of the full-text index. Their short labels were winning BM25 against the memories they came from and eating seed slots.
- Two small on-device models. A 384-dimensional embedding model (bge-small) as an extra retrieval leg, and a cross-encoder reranker (MiniLM). Both run through ONNX on the CPU. No network, no key. The reranker alone was worth 13 points at 5 memories (p = 2e-14), which is almost exactly what the hosted reranker was worth to MemVault when I measured that.
- Giving the reranker more to judge. The last diagnosis was the most useful one: on multi-hop questions, which need about three memories, the right ones were all inside the top 50 only 36% of the time, and when they were, the answer was right 90% of the time. The problem was the candidate pool, not the model. Letting the reranker score 100 candidates plus the memories saved right around the top hits, with the embedding leg weighted double, was worth another 3.7 points at 50 memories (p = 5e-4 on test) for a doubling of latency.
The final configuration, chosen on dev and confirmed once on the held-out test split:
| Test split, 885 questions | 5 | 10 | 20 | 50 | Recall |
|---|---|---|---|---|---|
| NSM, no model | 62.3 | 69.4 | 75.5 | 81.8 | 6 ms |
| NSM, on-device models | 76.5 | 81.8 | 84.6 | 87.1 | 1.3 s |
| Hosted embedding search | 52.0 | 60.6 | 68.5 | 77.1 | 308 ms |
| Mem0 cloud (776 questions it answered) | 75.3 | 81.1 | 85.6 | 468 ms | |
| MemVault production | 87.3 | 91.9 | 785 ms |
What I can say now, and what I cannot
I can say: a memory that runs entirely on your machine, with no account and no per-query cost, scores 86.9% against Mem0's cloud product's 85.6% at 50 memories on the questions both answered (p = 0.32, not a significant difference). With no model at all it still beats hosted embedding search at every size, in 6 milliseconds.
I cannot say it matches current MemVault. It is 2.7 points behind at equal context and 4.8 behind in high-context mode, both significant, and Mem0 at its 200-memory setting, with nearly three times the tokens, is ahead too. The gap is mostly multi-hop questions, where the small local models do not bridge paraphrase the way a 768-dimensional hosted embedding and a large reranker do. I cannot say the graph does anything. I cannot say "fast and accurate" in one breath: the 6 ms configuration and the 87% configuration are different configurations, and the second one takes 1.3 seconds.
I also cannot say the no-model improvements are proven. They were significant on dev and went the same direction on test, but did not reach significance there on their own. The honest statement is "about one point."
The part that matters more than the numbers
Every claim above has a run directory behind it, a paired test, and a line in a results file that also lists the rejected ideas. That file is the most credible thing in the repository, more than any single score, because it is the only thing that could not have been produced by an agent typing numbers into a table.
If you build with agents the way I do, here is the rule I am keeping: measured, never typed. An agent can write a benchmark harness. It can also write the harness's output. The only way to tell the difference is to look at the code that produced the number, run it yourself, and keep the losses in the same file as the wins.
NSM is open source and installs in two minutes: github.com/anthonylee991/nsm. Every number in this post, including the ones against it, is reproducible from the harness in the MemVault repository.