Doing that would mean copying and keeping in-sync two files: AGENTS.md and CLAUDE.md since I use both, claude, and others intercheangeably in one project. Also, I dont want to keep my project's details in those files, but, keep it separate. With current setup/way, a single line in both satisifies all constraints and requirements.
I use AGENTS.md as the "global" agent file, then use CLAUDE.md as a light wrapper with Claude-specific instructions which ends in an instruction to read AGENTS.md .
That works better. I just personally avoid touching global configs, hence, the project specific copy-pasting of two files.
So about a year ago I wrote my own attempt at something like this using vector indexing and BM25 (the latest version uses CocoIndex, I had a custom coded solution using ChromaDB before). I wrote a comprehensive enough test set that showed performance increases on the quality of search results and reduction in token usage versus grep and rg. I haven't had time to really polish it but it worked well enough, particularl…
Wow, thanks for sharing, and cool that you're working on similar things! Feel free to drop any feedback on the repo if you want!
What I have personally observed with such tools is that they make the AI's dumb, similar to how it makes coders dumb when relying more on AI tools. These agentic AI's are already smart enough to figure out a highly optimized path to code exploration or search. But, with these tools, they just go very aggressive, partly because the search results from these tools almost in 100% of the cases do not furnish full details…
I'm seeing over and over again people claiming absurd optimizations for coding agents: > Our tool uses 99x fewer tokens and delivers 88x better results. Okay, great, but... 1) It's VERY difficult to quantify something is better . 2) They almost never post how they measured how much better it is and what the margin of error might be. 3) I assume they are incompetent and don't even try the tool. Like you pointed out, t…
Hey, this skepticism is fair and we share it, which is why we don't claim end-to-end agent improvements since we haven't measured those (yet). The benchmark we published measures retrieval quality and token count during search, not overall agent performance. We are working on agent-level evals, but those are unfortunately much harder to get right. However, we do believe that Semble makes agents better based on our own experience of using it for the past months while in development (or at the very least, cheaper).
Interesting for sure, but I think I need to be convinced this is necessary. How many tokens does Grep use on the average response anyways? Does this reduce input token usage from 100k to 2k per query or from 1k to 20? How does this effect output code quality?
There's a "semble savings" command you can run to see token savings. It varies heavily based on the repo: in general, the larger the project the larger the savings. Output code quality is something we're still trying to measure, but it's much harder than measuring retrieval quality.
These almost always come with the cost of worse adherence
Fair, many tools trade off cost for performance/adherence. So far our experience is good with Semble (at least with Anthropic/OpenAI models), but if you have any feedback feel free to reach out. We're working on evaluations for this as well, though it will take some time as this is much harder than benchmarking retrieval quality.
Interesting for sure, but I think I need to be convinced this is necessary. How many tokens does Grep use on the average response anyways? Does this reduce input token usage from 100k to 2k per query or from 1k to 20? How does this effect output code quality?
There's a "semble savings" command you can run to see token savings. It varies heavily based on the repo: in general, the larger the project the larger the savings. Output code quality is something we're still trying to measure, but it's much harder than measuring retrieval quality.
Gotcha. Might be interesting to show some plots of average/median/max input token savings per query as a function of repo size if you're going to do some more comparison testing. The efficiency gains are compelling, but I'd want to see the magnitude of gains as well to get a full picture. Regardless, cool project and I'll check it out for myself
I'd be interested in seeing actual agent benchmarks (eg CC or Copilot CLI with grep removed and this tool instead). For example, I have explored RTK and various LSP implementations and find that the models are so heavily RL'd with grep that they do not trust results in other forms and will continually retry or reread, and all token savings are lost because the model does not trust the results of the other tools.
I think the best bet is to use some kind of proxy so when the model calls grep, you intercept the call, use other tool to search and give back the results to the model.
True. Just have the interface that behaves like grep and the output is as expected like grep but internally: indexed, ranked, ...
So the model trusts the output because it is grep :D