Better Memory Pipeline

better-memory

[test]

A nightly pipeline that turns a personal Obsidian knowledge base into memory any Claude can recall from: on a phone, a desktop or the web, without reading the vaults.

Matt Guerra built and runs it on his own vaults. The code, the schedule and the distiller are the real ones; the vaults in this repo are an invented sample (a fictional ceramics teacher, Rowan Adler), because the real ones are private by definition.

How it works

vaults (Obsidian, several roots)
   |
   |  gate.py        nightly at 20:00 (launchd). Not AI. The security boundary.
   |    - walls off private folders, dotfolders and traversal, fail-closed
   |    - skips vault scaffolding, keeps one explainer note per nested vault
   |    - hashes each safe note's post-gate content; stages only new or changed notes
   |  content_gate.py   optional second wall, on content instead of paths
   v
~/cowork-staging/staging/        flat files: vault__folder__note.md (tonight's delta)
~/cowork-staging/.manifest.json  the hash record that drives change detection
   |
   |  distiller      nightly at 20:30. A Claude skill run as a Cowork task.
   v
Notion "better-memory"           one page per entity, typed Relations: a knowledge graph
   |
   v
any Claude with the Notion connector reads it for recall; nothing else writes it

Try it

VAULT_ROOTS="$PWD/sample-vaults/workbench:$PWD/sample-vaults/identity" \
STAGING_DIR="$PWD/.staging" python3 gate.py
ls .staging/staging

python3 -m unittest discover -s tests

Python 3.10+, no dependencies.

The invariants that make it safe

• The wall is enforced by the filesystem, not by the model. The distiller is only ever given the staging directory, never a vault. The private folders are not "excluded by instruction"; they are physically absent from everything the model can read.
• The gate fails closed. Paths resolve before they are checked, so .. and symlinks out of a vault are refused. On startup the gate probes its own walls with known-private paths and refuses to run if any probe gets through.
• Staging is disposable. Every run wipes it and rebuilds it, so the distiller never reads yesterday's leftovers.
• Only the nightly task writes the database. Every interactive session reads it, so recall never races a live edit.

Design decisions

• Content-hash manifest, not a finished/ folder: an edited note re-stages and an unchanged one is skipped.
• Hash after the gate: only a change in what is actually staged re-stages a note.
• One explainer per nested vault: instruction files, indexes, templates and run logs are skipped, but each nested vault keeps one note saying what it is for.
• Section gate off by design: private material is already behind path walls, so its pattern lists are empty and an empty list matches nothing.

What's here