README.md

README.md

1. # better-memory
3. [![test](https://github.com/mattrguerra/better-memory-pipeline/actions/workflows/test.yml/badge.svg)](https://github.com/mattrguerra/better-memory-pipeline/actions/workflows/test.yml)
5. A nightly pipeline that turns a personal Obsidian knowledge base into memory any Claude
6. can recall from: on a phone, a desktop or the web, without reading the vaults.
8. Matt Guerra built and runs it on his own vaults. The code, the schedule and the distiller
9. are the real ones; the vaults in this repo are an invented sample (a fictional ceramics
10. teacher, Rowan Adler), because the real ones are private by definition.
12. ## How it works
14. ```
15. vaults (Obsidian, several roots)
16. |
17. | gate.py nightly at 20:00 (launchd). Not AI. The security boundary.
18. | - walls off private folders, dotfolders and traversal, fail-closed
19. | - skips vault scaffolding, keeps one explainer note per nested vault
20. | - hashes each safe note's post-gate content; stages only new or changed notes
21. | content_gate.py optional second wall, on content instead of paths
22. v
23. ~/cowork-staging/staging/ flat files: vault__folder__note.md (tonight's delta)
24. ~/cowork-staging/.manifest.json the hash record that drives change detection
25. |
26. | distiller nightly at 20:30. A Claude skill run as a Cowork task.
27. v
28. Notion "better-memory" one page per entity, typed Relations: a knowledge graph
29. |
30. v
31. any Claude with the Notion connector reads it for recall; nothing else writes it
32. ```
34. ## Try it
36. ```bash
37. VAULT_ROOTS="$PWD/sample-vaults/workbench:$PWD/sample-vaults/identity" \
38. STAGING_DIR="$PWD/.staging" python3 gate.py
39. ls .staging/staging
41. python3 -m unittest discover -s tests
42. ```
44. Python 3.10+, no dependencies.
46. ## The invariants that make it safe
48. - **The wall is enforced by the filesystem, not by the model.** The distiller is only
49. ever given the staging directory, never a vault. The private folders are not
50. "excluded by instruction"; they are physically absent from everything the model can
51. read.
52. - **The gate fails closed.** Paths resolve before they are checked, so `..` and
53. symlinks out of a vault are refused. On startup the gate probes its own walls with
54. known-private paths and refuses to run if any probe gets through.
55. - **Staging is disposable.** Every run wipes it and rebuilds it, so the distiller never
56. reads yesterday's leftovers.
57. - **Only the nightly task writes the database.** Every interactive session reads it,
58. so recall never races a live edit.
60. ## Design decisions
62. - **Content-hash manifest, not a `finished/` folder:** an edited note re-stages and an unchanged one is skipped.
63. - **Hash after the gate:** only a change in what is actually staged re-stages a note.
64. - **One explainer per nested vault:** instruction files, indexes, templates and run logs are skipped, but each nested vault keeps one note saying what it is for.
65. - **Section gate off by design:** private material is already behind path walls, so its pattern lists are empty and an empty list matches nothing.
67. ## What's here
69. | Path | What it is |
70. |---|---|
71. | `gate.py` | The path gate: walls, self-check, scaffolding rules, change detection. |
72. | `content_gate.py` | The optional content gate: frontmatter flags, line patterns, remove or redact, fail-closed. |
73. | `distiller/` | The distiller skill, its schedule and binding, and the Notion schema. |
74. | `scheduling/` | The launchd job and the nightly wrapper. |
75. | `sample-vaults/` | Two invented vaults that exercise every rule. |
76. | `examples/` | A real gate run on the sample, and one distilled page. |
77. | `tests/` | The test suite, stdlib `unittest`. |
78. | `docs/INSTALL.md` | Installing it on a Mac. |