Memory Needs a Way Out

For months, every AI assistant I work with had the same context: I was moving abroad by spring, and my work decisions should serve that move. Then the plan died. I told one assistant, but the others kept framing their advice around a move that wasn't happening, because the memory system that feeds them had no way to forget.

I fixed that in one night by measuring how bad the problem was and building a way to clean it up.

The System, and the Bug in Its Shape

My notes live in an Obsidian vault. Every night, a script called the gate walks the vault, walls off private folders, and copies new or changed notes into a staging folder. A scheduled Claude task, the distiller, reads staging and writes one page per person, project, or thread into a Notion database. Any Claude I talk to, on any device, can read that database for recall.

The gate was built to be careful about what goes in, but it had nothing to say about what should come out, and that caused two problems:

1. Deleted notes never left Notion. The gate compared each night's notes to the previous night's, but a note that disappeared just dropped out of the comparison. Its Notion page lived on as an orphan.
2. Change detection only looked at the words, and never asked whether they were still true. The gate only re-sent a note when its words changed. A note that said "planning X" and was never edited was never looked at again, no matter how false it had become.

So facts could get into the memory, but nothing ever took them back out.

Fix One: A Revision Path

The new gate, gate_v2, adds tombstones, volatility tiers, and a way to record how a plan resolved.

• Tombstones. When a note that existed last night is gone tonight, its path goes into a tombstones file, and the distiller archives whatever came from it. Tombstones are re-sent for seven nights, so one failed run doesn't lose them. A guard holds them back if more than a quarter of the vault vanishes at once, because that's much more likely to be a misconfigured path than a real cleanup.
• Volatility tiers. A note can say how fast its facts change: bedrock (never), stable (true now, with no clock on it), provisional-active (a live decision, reviewed every 7 days), or provisional-dormant (parked, reviewed every 90 days). A provisional note past its review date gets re-sent even if nobody touched it, and it's flagged as due.
• Resolution as data. When I close a plan, I don't delete the note. I add a resolved-to line in my own words. The distiller treats that line as the current truth and rewrites the old plan in the past tense.

That fixed the mechanism, but it didn't tell me which of my 4,500 notes were stale.

How Stale Is It, Actually?

My prediction was that most of the notes in my vault that describe plans are out of date. Before testing it, I locked in the rule: the prediction would hold if more than half of the plan notes in a random sample came back stale or dead, and anything that wasn't really a plan would drop out of the count.

I limited the test to eight folders of my own work (leaving out research archives), had a model find the plan notes (34 of 196), drew 20 at random with a fixed seed, and labeled each one myself.

13 of the 20 were stale or dead, which is 65%, so the prediction held. None of the 20 were false positives.

The process taught me more than the number did:

• The margin is wide. With only 20 notes, the true share sits somewhere between about 43% and 82%. It's honest to say "most of the plan notes I checked were out of date." It isn't honest to say "most of my plans are stale."
• The text can't tell you. The model's "is this a plan" score averaged 0.73 for dead notes, 0.74 for stale ones, and 0.79 for live ones. A note can be clearly a plan and still be completely dead.
• Neither can the date. The dead notes were the youngest in the sample. The oldest note, untouched for six months, had been parked on purpose and was still valid.

Since neither the note's text nor its timestamp gave staleness away, I started looking at what each note depends on.

A Small Model That Only Answers Yes or No

The classifier is TypeSafe's System One model, Jev. It doesn't write text. You give it a state (in this case, a note) and a set of questions, and each yes/no question comes back as a probability between 0 and 1. The questions run in parallel and can't see each other's answers. That makes it cheap and easy to inspect, but also very sensitive to how the questions are worded.

Here's what three rounds of tuning taught me:

• Ask about the whole note. "Does this note describe an open plan?" fired on any TBD. Asking "Is this note mainly about a plan not yet made?", along with written criteria for yes and no, cut the average miss on settled notes from 0.39 to 0.03.
• Frontmatter leaks answers. If the model can see status: active, it reads the label instead of the note. Every run now strips the frontmatter before sending.
• Several yes/no questions beat one multiple-choice question. A single "pick the tier" question did worse than three yes/no questions combined by code: is it a plan, is it parked, and would it stay true for years.
• Don't hand it the age. Telling it how many days had passed since an edit made it call everything dormant. The script still prints the age, but that's for me to read, and the model never sees it.
• Flag the scores near the middle. Scores near 0.5 get flagged for a human. In the final triage test, every note the model was sure about was right, and the one miss was one it had flagged as unsure.

The Dependency Graph

If staleness spreads through dependencies, the vault needs a map of which notes depend on which. Asking the model about every pair of notes in scope would have meant 17,578 pairs, so the first pass uses no AI at all.

A free filter. I kept only the pairs with some evidence of a connection: one note links to the other, or mentions it by name in plain text. The first version matched any title, and the word "pipeline" alone matched 88 times. Counting only multi-word names, matched as whole words, cut it down to 389 candidate pairs, which is 2.2% of the total.

Before any model ran, that pass turned up two problems.

The first was 18 places where a live note pointed at a note I had already closed. Staleness had visibly spread, and a simple link check was enough to see where.

The second was the note the whole problem started from. Every cleanup that night had searched for the city's name and fixed the notes that repeated the plan. None of them touched the project note that actually was the plan. It still said status: active, still had a target date, and five other notes named it. A word search finds copies of a fact. A dependency graph finds where the fact comes from.

Direction, decided by the model. For each candidate pair, Jev answered two questions: does A depend on B, and does B depend on A? I defined "depends" as a test: if B changed or stopped being true, would A need updating? Sharing a topic, or citing B for reference, doesn't count.

The full run took 389 calls and cost two cents, and the direction came through clearly. For example, each tailored resume depended on its master resume at 0.92 to 0.96, with the reverse scoring around 0.1. No pair scored as a dependency in both directions.

18 became 6. Of the 18 places where a live note pointed at a closed one, the model identified six as real dependencies. I checked all six, and all six were real and needed action. One rule came out of the labeling: a note that depends on a superseded plan is stale rather than dead. The subject lives on in a new form, so anything that borrowed from the old plan needs updating. My resume templates still said the portfolio wasn't built, and two of them were being held back as drafts, waiting on a site that had already launched.

Keeping a Human in the Loop

The model never writes a confirmed dependency. Instead, it writes depends-on-proposed into a note's frontmatter. A Templater script, bound to a hotkey in Obsidian, walks through each proposal. Accepting moves it to depends-on. Rejecting moves it to depends-on-rejected, so it's never proposed again. Skipping leaves it where it is.

89 proposals went in. I accepted 86 and rejected 2, and one went missing because I edited that note by hand. Since nothing marked it as rejected, it'll simply come back on the next run.

A 97% acceptance rate can be read two ways. Either the model is precise at the cutoff I chose, or a 56-note queue late at night makes it easy to just keep clicking accept. This run can't tell those two apart, and it never measured what the cutoff missed. Re-reviewing a random ten in a fresh session, and sampling the 122 middling scores I never saw, would answer that. Until then, the honest claim is precision on a small sample, with recall still untested.

What I'd Tell Anyone Building AI Memory

• Give memory a way out. If facts can get in but never leave, the system will keep describing a version of you that's out of date.
• Make resolution a field instead of a deletion. A closed plan is still information. A resolved-to line in your own words tells every downstream reader what happened, instead of leaving a gap.
• Don't rely on text or timestamps to find staleness. Both failed here, and dependencies worked.
• Spend the AI where judgment is actually needed. A free link scan narrowed 17,578 pairs down to 389. The model only judged direction, and a human only confirmed.
• Write your predictions down first, then report the misses. I recorded my predictions before each run. Several of them were wrong, and those are in the build log too.