Persistent-memory evaluation (write, recall, staleness)

How it works

A system that remembers across sessions carries a second, slower state store that nothing else in the harness evaluates: written by the model, read back by the model, and in practice never reviewed by a person. Testing it needs a unit larger than a request — scripted multi-session episodes, where the first session establishes facts, a later one changes or contradicts them, and a third checks what the system actually does. Inside that frame each step gets measured on its own terms. Write precision and recall: was the durable thing saved, was the ephemeral thing left out, was an existing entry updated rather than a near-duplicate created. Fidelity at the moment of writing, which is restatement fidelity applied to a note the system will later treat as established fact. Retrieval at the right moment, which is retrieval evaluation against a corpus the system wrote itself — and where a false positive hurts more than a miss, because an irrelevant memory arrives carrying the authority of something already settled. Staleness, because an entry records what was true when it was written and nothing in the store knows the world moved on. And correction and deletion: whether a wrong entry can actually be fixed, whether the fix takes effect on the next read, and whether a deleted one stays gone. What earns this its own suite rather than a footnote under the others is compounding — an error written once is read back as fact in every later session, by a system with no reliable way to tell what it was told from what it inferred.

When to use it

Any assistant or agent with memory that outlives the session; before letting a system write its own memories unreviewed, and again after changing the instructions about what it should save; where memory is shared across users, projects or agents and one context's note can surface in another; and as a standing check rather than a launch gate, because the store only degrades with age — the oldest entries are both the most likely to be wrong and the least likely to be re-read by anyone.

Limitations

Scripted multi-session episodes with known end states are heavier setup than any single-request method, and they still won't reproduce what real users establish incidentally over months. Write precision rests on a judgement about what deserved saving at all, which is product policy rather than ground truth — two reasonable annotators disagree, and the right answer differs per system. Staleness is only measurable against facts you decided in advance to invalidate, so the slow drift of something nobody flagged stays invisible by construction. And a clean episode suite says little about the store as it actually ages in production, where the entries are the ones the system chose to write, not the ones a test author imagined.

Cite this