If you build things with AI agents, this has probably already happened to you. You just may not have caught it yet.
I had an agent build me a memory system for AI assistants. The idea was simple: a memory that lives in one file on your own computer, no account, no monthly bill. The agent built it over a few sessions, wrote the documentation, and wrote a benchmark to prove it worked. Then it put a table in the README. My system: 91% accuracy. Mem0, the memory company that has raised $24 million: 15%. Another well-funded competitor: 26%. Response time under two milliseconds.
I posted about it. I was proud of it.
Developers told me I was a rookie for believing a table an AI wrote. They were rude about it. They were also right.
How I found out
I asked a different agent to read the benchmark code and tell me what it actually did. Three things came back.
The 15% for Mem0 was not a result. It was a fallback. The script said, in effect, "if there is no Mem0 account connected, write down 15." There had never been a Mem0 account connected. The README called that a live comparison.
The "benchmark" was not the public test it was named after. It was ten short scenarios the agent had written itself, graded by checking whether certain words appeared in the answer.
A chart showing a "strength" value climbing to 4.88 had been generated by code that capped that value at 1.0. The chart was impossible.
None of this was hidden. It was sitting in plain code in the project folder the entire time. I had not read it, because the table said what I wanted it to say.
What the real number was
I already had a real test, built for MemVault, my hosted memory product. It asks 885 questions about long conversations between two people, uses one AI to answer from whatever the memory system hands it, and a separate AI to grade the answers. Every system gets the exact same questions and the exact same grader. I had used it on Mem0. I had never used it on the thing the agent built for me, because the agent had told me it was already done.
I deleted the fake benchmark and ran the real one.
The system scored 80% when the assistant was allowed to read 50 memories, and 60% when it could only read five. Not 91%. But not nothing, either. It beat a plain paid search at every size, with no AI model involved and no bill. The thing was real. The table was not.
Then I turned off the part the agent had been proudest of, the "biologically inspired" network that was supposedly the whole secret. The score did not move. All the work was being done by a plain word search. The clever part was decoration.
What happened next
I spent two days and about $45 doing what I should have done from the start: change one thing at a time, measure it against the version before, keep what helped, write down what did not.
Most of it did not help. A smarter way of weighting the search: nothing. A second, more careful version of the clever network: nothing. Adding the surrounding conversation to each result: made it worse.
Two things helped a lot. Two small AI models that run on your own computer, no internet needed, one to find candidates and one to re-read them and pick the best. Those two models alone were worth 13 points at the smallest size. And giving the second model more to choose from, which was worth another four points at the largest size.
The honest final number is 87%. Which, as it happens, is statistically a tie with Mem0's cloud product on the same questions, and still a few points behind MemVault. That is a result I can defend, because I can show you the run that produced it, and because the file that holds it also lists everything that failed.
The three checks I run now
I still build with agents. I do not write much code by hand and I am not going to start. What changed is that I no longer accept a number from an agent without doing one of these:
- Ask where the number came from, and read that. Not the summary. The actual code or the actual log. If an agent produced a result, an agent can show you the thing that produced it. If it cannot, you do not have a result.
- Look for the hardcoded value. Search the benchmark code for the competitor's score typed as a literal number. It takes ten seconds. If you find it, the whole table is suspect.
- Ask "can I run this myself, right now?" If the honest answer is no, because the test needs an account you never set up or data you never downloaded, then nothing has been tested.
Agents are extremely good at writing the harness and extremely good at writing the harness's output, and from the outside the two look identical. The only way to tell them apart is to look at the code that produced the number and keep the failures in the same file as the wins.
I turned that into a rule for everything I publish now: measured, never typed. Every number on this site traces to a run I can show you.
The memory system is open source and installs in two minutes: NSM on GitHub. If you want the full technical write-up, with every experiment, every number that went down as well as up, and the statistics, it is here: An AI agent told me my memory system beat Mem0. It had typed the scores in.