61–70 of 97 posts
Tools that remove the fat seem like a good idea, but I’m highly suspicious of their effect on the LLM’s reasoning. LLMs were trained in the typical full-fat output found everywhere on the internet, and all of sudden they get a slightly different response that may look like nothing they have seen before. Does that really save tokens in the long run?
I would like to have deeper comparison with alternatives like rtk, which are already fast and written in rust, also the previous comments mentioned something that has been a know problem with rtk that it sometimes strips the thing that the llm needs (or expects, causing more work to need to happan not less)
None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…
There is definitely tons of value to extract from this line of thinking.
I have my own llm wrapping harness, which does this and has a few more tricks. For example, it doesn’t have a lot of mcp but it does have search_mcp and load_mcp tools (and search_skills) so the llm can find what it needs when it needs it without bloating the normal baseline context. The LLMs have proved really good at using them. There is also a waypoint tool they can use to record their thinking in the context with…
Earlier quoted context omitted.
None of these tools measure how effective they are... It's a massive red flag to me when you could get decent data to see if your thing actually works, and they don't even attempt to... Have the LLM use your tool, run it on several of the coding benchmarks. If you're stingy, run it on the ones that don't cost much. Otherwise, I'm going to assume it doesn't actually work. If it did - Claude, Antigravity, Codex, Pi, or…
> otherwise, popular solutions would integrate the idea None of the major players are incentivized to care about this, especially not over other opportunities. Why would you expect them to integrate it? One of the biggest wins you can institute for your own codebase if you use agents is writing your own harness, by a huge margin. The defaults are fine, but you can do better.
Earlier quoted context omitted.
I have a suspicion that the model would miss more context unless you are very precise about what FAT means in each context. However, loved the idea.
Understood. Let me give some examples, most of the time we don't need spaces between table output, git diff produce bunch of unnessary info we just need filename and actual diff lines, kubectl describe we would mostly check for events, image etc etc. This is the reason why I make it as composable filters as it very depends on your specific ops to optimize the token.
Harness: I'm about to commit. Good use case Harness: What has changed from X to Y. Bad use case NO?
Earlier quoted context omitted.
This is the reason, when I built a tool in the same space, I chose to benchmark with cost per correct answer. Reducing tokens and also turns is quite worthless if the LLM doesn’t solve what you put it to do.
Did you benchmark the competition and can we see?