I don't understand how this or all these magic skill bundles and methodologies get traction and why they are so popular. It's either plain worse or has serious trade offs.
RTK reports token savings, but our cost benchmarks disagree
11–20 of 85 posts
Re: RTK reports token savings, but our cost benchmarks disagree
#12well yeah.... now you're giving it output it wasn't trained on.
What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)
Re: RTK reports token savings, but our cost benchmarks disagree
#13i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
I don't think they're comparable. RTK just modifies the output of CLI tools to reduce the number of tokens, a Roslyn MCP gives the agent a fundamentally superior way of interacting with a C# codebase.
My main issue with rtk is that rtk randomly messes modification and agent start polling same tool continuously.
Re: RTK reports token savings, but our cost benchmarks disagree
#14Just another instance of the bitter lesson. The model itself knows how to be clever and conserve tokens in command output by using shell primitives and as the models get smarter they get better at anticipating large output and defensively adapting the input commands.
Re: RTK reports token savings, but our cost benchmarks disagree
#15If you are using maven you should tell your agent to use its quiet mode or rtk, because mvn love to write a lot of useless output.
Re: RTK reports token savings, but our cost benchmarks disagree
#16I don't understand how this or all these magic skill bundles and methodologies get traction and why they are so popular. It's either plain worse or has serious trade offs.
It is just "putting a wet phone in rice" of AI
Re: RTK reports token savings, but our cost benchmarks disagree
#17It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?
My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark.
Here are other cases demonstrating the exact same issues with these kinds of tools:
https://blog.jetbrains.com/ai/2026/07/rtk-claude-code-token-... https://brandonbarker.me/writing/headroom-fewer-tokens-bigge...
Re: RTK reports token savings, but our cost benchmarks disagree
#18All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…
I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours?
Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec.
Releases v0.0.42 Latest last month
Re: RTK reports token savings, but our cost benchmarks disagree
#19[flagged]
Re: RTK reports token savings, but our cost benchmarks disagree
#20All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…
Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.