i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
Which one specifically? The ones I found looked like unmaintained throw-away experiments.
RTK reports token savings, but our cost benchmarks disagree
51–60 of 87 posts
Re: RTK reports token savings, but our cost benchmarks disagree
#52All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…
I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only…
I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days.
But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity.
The caveman gimmick makes sense to me as a clever hack. Caveman talk is a longstanding meme that's presumably well-represented in the models' training data. So just asking it to do that is just an ultra-concise way to tell the LLM to be ultra-concise. Which, in turn, is theoretically good for accuracy because putting too many instructions in the prompt is bad for task performance.
Re: RTK reports token savings, but our cost benchmarks disagree
#53The technique looked dubious from the start, because LLMs were trained to expect certain outputs from common bash tools. If the output is not what it expects, an LLM may issue more tool calls than before, because it will assume the tool is broken, the arguments passed to it were wrong, or it's a newer/older version of the tool etc => more tokens. Sounds like just adding to the prompt to use `grep` and `tail` extensiv…
This was the problem I saw. I installed rtk when it came out and liked the idea of it. But over time with newer model generations I kept seeing the model get confused in the reasoning text and retry a command bypassing rtk. I didn't even need a benchmark to see it was regularly an impediment to the final outcome.
Re: RTK reports token savings, but our cost benchmarks disagree
#54[flagged]
Re: RTK reports token savings, but our cost benchmarks disagree
#55All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…
[flagged]
Re: RTK reports token savings, but our cost benchmarks disagree
#56I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.
Re: RTK reports token savings, but our cost benchmarks disagree
#57Re: RTK reports token savings, but our cost benchmarks disagree
#58It's pretty damn obvious to anyone who ever bothered to look at rtk gain output, no benchmark needed at all. Agent runs
rtk command-that-prints-100k-tokens | tail -5
Worse, since rtk defaults to persisting that savings stat, it breaks sandboxing. Prefixing with rtk leads to random auto-mode denials from time to time too (this is independent of disabling savings stat persistence).
Honestly have no idea why anyone who knows the first thing about CLIs would take rtk gain seriously. I guess clueless vibe coders who has hardly ever worked in a terminal before will look at the stat and feel good about it?
That said, rtk is still mildly useful for compressing repeated test run outputs and stuff, but you should only ever use it on whitelisted commands; wrapping everything like they suggest you to do is just stupid.
Re: RTK reports token savings, but our cost benchmarks disagree
#59It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…
Even JetBrains is now AI blog-slop, how disappointing.
Not all, but definitely on some. I also find it very off-putting.
Re: RTK reports token savings, but our cost benchmarks disagree
#60This is my first time hearing about RTK, and yikes! The benchmarks mean nothing; this thing is actively dangerous to use.
If my agent runs a shell command, show my agent the output of that shell command. I don't have a problem with automatically pruning or paginating large outputs, as long as the agent still has some form of access to the original output (e.g., by searching).
But removing verbosity from an 'ls' command? That's ridiculous! If my agent runs `ls -la`, do not drop the owner and date from the output, because THAT'S AN EXPECTED PART OF THE OUTPUT. Nothing should be trying to predict the agent's intent and interfering, unless you want your agent to get confused and fail.
These LLMs were trained on predictable shell behavior, and RTK deliberately subverts the model's expectations. There's no way that isn't degrading the model's capability.