If you build with AI agents, you have run into the memory problem. You tell the agent something on Monday, and on Wednesday it has no idea. Every tool that promises to fix that wants a subscription, an API key, and a database.
So the advice going around right now is appealing: skip all that. Have the agent write what it learns into plain text files, the same kind you could open in Notepad. When it needs to remember something, have it search those files for the right words, using the search command every computer already has. No account. No bill. Instinct, the agent company that just raised at a ten billion dollar valuation, reportedly works roughly this way under the hood, and that has made the idea feel settled.
I wanted to know if it actually works, so I tested it. This post is what I expected, what I found, and what to do about it. The short version: the files are fine. The search is the problem, and the fix is free.
How I tested it
For the last year I have been running memory systems through the same exam. It is a public dataset of long conversations between two people, spread over months, with 885 questions about what was said: "What did she say her brother does?", "When did he start the new job?", "Which city were they planning to visit?" Every memory system gets the same conversations, the same questions, the same AI answering from whatever the memory hands it, and the same AI grading the answers. I have used it on paid products like Mem0, on my own hosted product MemVault, and on my free local one, NSM. The scores are comparable because nothing changes between systems except the memory.
For this test I built the text-file setup exactly the way people describe it. Each conversation session became one text file, with a dated line for every message. Nothing clever. Then I had the "agent" do what an agent with a terminal does: take the important words out of the question and search every file for them.
I tried three ways of reading what the search found, because that turned out to be the whole story:
- Read the matches in the order the search returns them. This is what grep does by default: matches appear in the order they sit in the files. It is what an agent reads when it runs one search and takes the output as it comes.
- Read the matches that contain the most words from the question first. Same files, same matches. The only difference is sorting them before reading.
- Read the whole note files, not single lines. Find the files with the most matching words and hand the agent all of them.
I ran each version at a few sizes, from "read five lines" up to "read fifty lines," or one to five whole files. Total cost for the test: about seven dollars.
What I expected
Honestly, I expected plain files to lose badly. Memory products use AI models to understand that "my brother" and "her sibling" mean the same thing. A word search cannot do that. I thought the gap would be big enough that the whole trick would look like wishful thinking.
I also expected the three ways of reading the results to come out about the same. They are reading the same matches, after all.
I was wrong about both.
What I found
Plain files with a word search are a real memory, if you sort the hits. Reading the lines with the most question words first scored 61% when the agent read five lines, 69% at ten, 74% at twenty, and 78% at fifty. For comparison, a paid embedding search, the "proper" way to do this with a hosted AI model, scored 52%, 61%, 69% and 77% at the same sizes. The free text files beat it at every size but the biggest, where they tied. They also matched my own NSM running with no AI model at all.
Grep's own order is not a memory. The same files and the same matches, read in the order the search spits them out, scored 27% at five lines, 38% at ten, 49% at twenty, and 64% at fifty. That is 15 to 34 points worse than the sorted version, on identical information. Nothing changed but the order the agent read the lines in.
Think about what that means. If your agent writes good notes and then reads the first few search results it gets, it is working from the wrong lines most of the time. Your notes are fine. Your agent is reading the wrong part of them.
Reading whole notes works best, at a price. When the agent picked the five best-matching files and read them end to end, it scored 83%. That is as good as Mem0's paid cloud product at its normal setting. The price is context: those five files are about two and a half times as much text as fifty lines, and every token the agent reads costs money and attention.
Where it stops working. Every version of the text-file approach falls down on the same questions: the ones where the answer needs two facts from two different places, or where the question uses different words than the conversation did. "What does her sibling do?" when the notes say "my brother is a nurse." The best text-file setup got 57% of those right. NSM with its small on-device models got 72%. MemVault got over 80%. A word search cannot bridge a paraphrase, and more files only help when the two facts happen to sit in the same one.
What to do about it
If you are using text files for your agent's memory, keep them. They work. Change how the agent reads them:
- Tell the agent to search each important word separately and count matches per file, then read the files or lines with the most matches first. In Claude Code or Cursor that is one instruction in your project rules file. It is the difference between 27% and 61% at the small sizes most agents actually use.
- Have it read whole notes, not snippets, when context is not tight. One good note beats twenty scattered lines.
- Keep your notes short and dated, one file per topic or per day. Everything here assumes the files were tidy. Nobody has measured a giant one.
If you want the version that handles the paraphrase problem too, that is what the small on-device models in NSM are for. It installs with one command, runs on your machine, and scored 87% on this same test with no account and no bill. This test is also how I know the models are the part of NSM that matters: switch them off and it is, more or less, a well-sorted word search with a few extras worth about three points.
What I did not test
Instinct's setup reportedly has one more step: once a day, an AI rewrites the raw notes into tidy fact files with a list of alternate words so searches find them. I measured the version people actually copy, raw notes and a word search, not that cleanup pass. I also tested one conversation's worth of files at a time, 19 to 32 files, not thousands. Both of those are fair follow-ups and I will say so plainly if someone runs them.
Every number here comes from a run I can show you, with the losses in the same file as the wins. The technical write-up with the statistics is in the MemVault benchmark results, and the code for the text-file setup is in the same repository as every other system it was compared against. If you think I got the setup wrong, the fastest way to show me is to change it and run it.