31–40 of 73 posts
> We evaluate on a 116-question representative subset of the LongMemEval benchmark (Wu et al., 2025), which tests an agent’s ability to answer questions over long conversations spanning multiple sessions.
I have always used traditional grep to search codebases. It serves me better than an IDE when there’re lots of scattered and frequent queries. grep’s design is surprisingly winning, exceeding expectations to this day.
pretty fast and neat project to search code interactively with a lot of optimizations on finding the right thing
This has been posted before, but a dead-simple pattern that helps enormously with steering the model to the right code area is a DESIGN.md that it creates, updates, and references periodically.
If you are truly bitter-lesson pilled - give the agent all the tools and let it decide which to use. - regex (grep) - hybrid search (bm25+vector) this X vs Y is uninteresting when the answer can be both.
I recently watched the new Palantir + Kirkland & Ellis fund formation platform demo, and I was surprised to see how effective the union of structured data was in an agent harness. We're used to dealing with flat files and comparing here basic ways of searching, essentially, long strings, but using Palantir's "Ontology" graph framework, I think Kirkland is going to be able to achieve some exception and differentiating…
> But it would make no sense to have an LLM regurgitate an existing form document token-by-token rather than call a piece of 1994 software like Hotdocs to populate some placeholders.
This is a real “oof”, isn’t it. Very difficult to understand what they were going for here. Perhaps they just assumed no one in the intended audience would pick it up. But it certainly is enough of a red flag that it made me go back to the top of your write-up for a re-read, thinking about their whole pipeline in much more sceptical terms.
Don’t presume this study has anything to do with programming. They measured an agent’s ability to search long conversations, not code. > We evaluate on a 116-question representative subset of the LongMemEval benchmark (Wu et al., 2025), which tests an agent’s ability to answer questions over long conversations spanning multiple sessions.