Memory Is a Pipeline, Not a Database: Designing AI Assistant Memory

Ask most people how to give an AI assistant a memory, and they’ll describe a database the assistant writes to. It learns something, stores it, and remembers it next time. That’s the design almost every product ships, and it’s the design I decided against.

Mine is a pipeline. I write notes in a vault. Every night, a job distills them into a database with one page per thing I might ask about. Any assistant I use, whether it’s on my phone, my desktop, or inside a coding session, can read that database when I ask about my own life or work. None of them can write to it. When I’m wrong about something, I fix the note, and the memory catches up by morning.

Why Not a Database the Assistant Writes

I had three reasons, and each one makes the others worse.

It accumulates its own mistakes. An assistant that writes down what it learned also writes down what it misread. A wrong inference stored on Tuesday is a confident fact by Friday, and nothing in the loop ever checks it against the source, because the memory has become the source.

It can’t disagree with itself cleanly. When two sessions learn contradictory things, both get stored. The next reader gets both and picks one. A memory with many writers has no single place where a contradiction gets resolved.

It has no gate. Whatever the assistant saw, it can store. A private detail mentioned in passing ends up in the memory, sitting in a vendor’s database, without anyone ever deciding it should be there.

A pipeline fixes all three by having only one writer. That writer is a job I control. It reads only what a gate let through, and every page it writes traces back to a note I wrote. So when something’s wrong, there’s only one place it can come from and only one place I have to fix it.

The Pipeline

The pipeline has four stops, and only one of them writes to the database.

A pipeline from vault to gate to distiller to recall layer, each box annotated, with a single write arrow into the recall layer. Sessions on phone, desktop and in code read on demand and never write, and a correction arrow returns to the vault to be re-distilled that night. — Figure 1 The write arrow only appears once, and that's the core of the design.

The vault is Markdown on my machine, and I write it. It’s the source of truth, and it stays that way no matter what any assistant thinks it learned.

The gate is deterministic code. It strips out the folders that are truly private, and it keeps a manifest of what it has already staged, so an unchanged rerun stages nothing and an edited note stages only itself. On the sample vaults on my site, it stages twenty-one notes the first time, nothing the second time, and one after a single edit. If I could only keep one test, I’d keep that one.

The distiller runs nightly. It reads the staging folder and nothing else, and it writes the database and nothing else. It’s model-driven, because turning a folder of notes into a page a stranger could use is a reading job. It’s also idempotent by title, which means a page gets updated in place and never duplicated.

The recall layer is the database. It has one page per entity, and it’s read-only to every session.

What a Page Looks Like

An entity is a person, an anchor, a thread, a context, or a note, and every page has the same shape.

A mock-up of one entity page from the recall database, for the pipeline itself: typed properties (title, type, derived from, distilled date, tags), a recall paragraph, observations tagged context, decision, pattern and constraint, and typed relations such as part_of and maintains. — Figure 2 The page for the pipeline itself, as it stands in the database.

The recall paragraph is written for a reader with no context. It says what the thing is, and then how to help usefully, which is a different question and the one an assistant actually needs answered.

The observations are tiered. Each line is tagged with what kind of fact it is: context, decision, pattern, constraint, status, or goal. That way an assistant reading a page can tell a decision I made apart from a pattern the distiller noticed, and treat them differently.

The relations are typed: part of, depends on, relates to, maintains. They’re what make the database a graph, and one hop in any direction is a query.

Title         Better-Memory Pipeline
Type          note
Derived from  STATE, me, better-memory-skill
Distilled     2026-09-19

[decision]    never a write target from an interactive session
[pattern]     idempotent by Title; update in place, no duplicates
[constraint]  privacy handled upstream at the gate

part_of    -> ideaverse
relates_to -> Daily News Briefing

That's the page for the pipeline itself, as it stood in the database the night I wrote this.

The Read Side

The rule is the same for every assistant I use. When I ask about my own life, work, people, projects, plans, or history, it looks in the recall layer first, before relying on any impression from past chats and before guessing. If there’s a page, that page is the authority. It shouldn’t query the layer for general questions, and it shouldn’t pull it proactively at the start of a session.

When I ask how it knew something, the honest answer names the source and offers to pull the live note so I can check it. That offer is important, because if I can’t trace a memory back to where it came from, I have no real reason to trust it.

Staleness Is Part of the Design

The recall layer can be up to a night stale. I know that, every assistant is told that, and it changes how the answer gets phrased. “As of last night” is something I can state precisely. With a live memory that may or may not have absorbed this morning’s conversation, I can’t say anything that precise.

Corrections follow the same path as everything else. A session that learns I was wrong doesn’t fix the memory. It tells me, I fix the note, and the pipeline re-distills it that night. The memory only has one author, and that author isn’t the assistant, so a contradiction only has one place where it can be resolved.

That’s how it’s designed to work. But the gate only stages a note when its text changes, which means a note that quietly stops being true, without anyone editing it, never gets looked at again. I found out what that costs when a plan I’d canceled kept showing up in every assistant’s advice. That story gets its own post.

Retrieval on Top

Keyword and property matching get you a long way with a few hundred well-written pages. The layer I’m building on top is for everything else: finding an entity by meaning when I can’t remember what I called it.

A flow from a question on the phone into two parallel searches, dense (a local 1024-dimension embedding) and sparse (a full-text index), fused by reciprocal rank fusion with K of 60 into one ranked list, then a one-hop walk along relations that returns the entity with its neighborhood. — Figure 3 Hybrid retrieval, then one hop. Done means it beats the keyword baseline on my own eval set.

Each page is embedded locally with a multilingual model, so Spanish and English land in the same space, and stored in a file-based vector store next to a full-text index. A question runs against both, the two rankings get fused using reciprocal rank fusion, and then the top entity’s relations are walked one hop so the answer comes back with its neighborhood. The vector search finds the right entity, and the graph supplies what’s connected to it.

The boundary holds here too. The indexer reads the staging folder, the manifest, and the database, but never the vault, and the embeddings never leave the machine. The entities table gets rebuilt in full every night so deletes are honored. Done means the hybrid search beats the keyword baseline on an evaluation set I wrote myself, and no vault read happens anywhere in the path.

What It Costs

The design costs me a night of latency and a gate I have to maintain. When the distiller gets something wrong, it has to be re-run rather than patched. It also takes the discipline of writing things down in the vault instead of telling the assistant and hoping it sticks.

I think that last cost is really a benefit. The assistant remembers what I chose to write, in my own words, in a file I own. To me, that’s closer to what memory means than anything a model decides to keep about me on its own.