31–40 of 41 posts
I don't get it. Why benchmark the latency instead of recall/precision? Optimizing for millisecond-level latency is meaningless in the context of LLM calls. Accuracy is the tool's greatest value, yet there is no testing for it?
Personally I've always seen AI 'memory' as a pain point for people in their experience using LLMs than a benefit from the agent remembering the last unrelated thing you were working on. It wastes context similarly to 'skills'. The most efficient workflow imo is having a few well written (not by ai) md files across a clean codebase.
To avoid losing context, I mainly conduct the planning session and the implementation session separately.
From the standpoint of building enterprise products, what worries me most is whether the agent we are implementing may not have understood a completely different context.
If okf_memory maintains domain knowledge very well, it is expected that implementation will be possible in unit functional units within a consistently smooth session.
However, there is a risk in applying this idea directly to practical work, so I’ll have to test it separately on a personal project.
I don't get it. Why benchmark the latency instead of recall/precision? Optimizing for millisecond-level latency is meaningless in the context of LLM calls. Accuracy is the tool's greatest value, yet there is no testing for it?
Has anyone else benchmarked all these tools for precision/recall? I too want to know if agent memory is something I should add. I only do session memory for now and that is quite useful.
But precision/recall is relatively "solved". What nobody has gotten close to solving is maintenance and provenance - what goes into memory, what qualifies as truth, how stale memory gets invalidated/superseded.
We're now in the phase of re-discovering 30+ years of pain of knowledgebases.
I want this, but also for cross-project memory. Save me from building my own, which I have planned but figure something would eventually pop up in HN...
I don't get it. Why benchmark the latency instead of recall/precision? Optimizing for millisecond-level latency is meaningless in the context of LLM calls. Accuracy is the tool's greatest value, yet there is no testing for it?
Has anyone else benchmarked all these tools for precision/recall? I too want to know if agent memory is something I should add. I only do session memory for now and that is quite useful.
deja: 29s to index, 24ms query, 18/100 hit@1, 67 found@50. Plain BM25, no vectors
agentmemory: 95s import, 14 hit@1, 65 found@50, plus a worker and engine on four ports
MemPalace: ~3h mining, 2.6s query, 14 hit@1
CASS: 56m index, every NL query fails with "query fuel exhausted" on the release build (fixed on their main)
claude-mem: no-op out of the box, only records forward from install
funes: the documented 1 min first pass indexed 189 of 19k sessions (0/100); full index still embedding, ~3 sessions/s
Numbers look low because 19k sessions is brutal; on the standard 500-session LongMemEval-S the same BM25 gets ~85% hit@1
The funny thing is BM25 basically ties embeddings here at 1/100th the cost. The real cliff is reranking (found@50 67 vs hit@5 35) and staleness. Vector search has zero concept of "superseded info" only fix I found was letting explicit user corrections outrank the transcript.
Repro scripts and corpus: https://vshulcz.github.io/deja-vu/guide/day-zero.html
@skeledrew: cross-agent across 23 harnesses, but yeah, it's an index over logs, not a source of truth :)