For someone who isn't super familiar, what is "R@10", and is 0.89 good? It's impossible to google for
89% chance the thing you want is among the 10 items returned by the system
So over 10% of the time, it fails? That's not a great search engine
We built a persistent agent memory layer on Elasticsearch with 0.89 recall
21–30 of 55 posts
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#22Earlier quoted context omitted.
I don't treat memory like RAG. That's the key. I only track decisions, actions, and outcomes.
Ah so you extract decisions, actions and outcomes and you index and search over them?
Yeah, after I tokenize them and embed them into vector form. Then it’s a simple cosine distance.
The point about memory is sometimes you remember great detail, sometimes you only remember that the memory exists, so having a good tool loop to attempt to recall and try permutations is good.
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#23This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate. This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
If you already have Elasticsearch, it makes sense to continue utilizing it. Saying, “just use SQLite” completely dismisses the idea that this is a _shared_ memory across teams. The ability to easily connect to the remote service and have everything “just work” pays dividends when you have dozens or hundreds of users.
I’m literally laughing at the root comment’s idea of proposing we replace ES with SQLite and imagining how that architecture review would go. Not everyone is doing MB/GB scale workloads.
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#24Earlier quoted context omitted.
89% chance the thing you want is among the 10 items returned by the system
So over 10% of the time, it fails? That's not a great search engine
It's sort of the whole tension when you query vectorized data/embeddings -- you need to balance accuracy and recall against the performance you need.
It took me a while to wrap my head around the two terms since they seem similar -- but Accuracy is basically "did i get mostly good results" and Recall is "did I get most of the good results" and they're subtly different. :)
Those two terms, though, will unlock as deep a rabbit-hole as you'd like on the subject.
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#25This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate. This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
Would be interesting if one can replace ElasticSearch with something like Typesense here
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#26Earlier quoted context omitted.
If you already have Elasticsearch, it makes sense to continue utilizing it. Saying, “just use SQLite” completely dismisses the idea that this is a _shared_ memory across teams. The ability to easily connect to the remote service and have everything “just work” pays dividends when you have dozens or hundreds of users.
I’m literally laughing at the root comment’s idea of proposing we replace ES with SQLite and imagining how that architecture review would go. Not everyone is doing MB/GB scale workloads.
that would be a pretty frail architecture too, I think I recall ES even saying not to rely on it for data persistence. Every time I've worked with ES it was always backed by some other database used as a source of truth.
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#27This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate. This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
Nah, "Any other vector DB" starts to fall apart once you need stuff like scripted scoring like OP uses. Then it starts to be a question of, "do you need ANN for performance?" since SQLite only does brute-force vector scoring. And granted, brute-force is performant for far more vectors than most people give it credit for, but it definitely hits a wall well below 1 million if you want it to have webpage-type latency. M…
it's also an odd situation to say a tabular database can replace a document store .. sure, it can, but that's not good practice from my point of view
also, I've run ES on an old laptop and it worked really well, so the cost of it can be pretty low if you're still in development
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#28I think this is cool and helpful but my biggest complaint is the writing style and word choice just scream LLM
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#29For someone who isn't super familiar, what is "R@10", and is 0.89 good? It's impossible to google for
https://www.evidentlyai.com/ranking-metrics/precision-recall...
"Good" is subjective.
Re: We built a persistent agent memory layer on Elasticsearch with 0.89 recall
#30This is such a basic thing nowadays, and ElasticSearch is massive overkill for it. Something like SQLite or LanceDB or basically any vector database is much more appropriate. This seems to be coming from the “we must make ElasticSearch AI-compatible” department more than anything.
I agree for casual usage, but this seems targeted towards enterprise setups, which makes much more sense to use something like ElasticSearch if you're already in the Amazon cloud, and especially if using the advanced features it provides like they are.