RTK reports token savings, but our cost benchmarks disagree
1–10 of 87 posts
Re: RTK reports token savings, but our cost benchmarks disagree
#2well yeah.... now you're giving it output it wasn't trained on.
Re: RTK reports token savings, but our cost benchmarks disagree
#3All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md.
What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly. Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt.
Why does it work you may ask? Well, LLMs basically brute force words/phrases and pipe that into find/grep/pgrep/whatever (or as recently discussed here write a python script for it - https://news.ycombinator.com/item?id=49654229). Semantic search looks for similarities so you have to do less brute forcing. Comes of course at the cost of indexing everything first.
You can find the project here: https://github.com/ory/lumen
Re: RTK reports token savings, but our cost benchmarks disagree
#4i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
Re: RTK reports token savings, but our cost benchmarks disagree
#5This makes sense. “Don’t try to penny-pinch your employees” is a lesson most managers learn eventually, and I guess agent-orchestrators will have to learn it too.
Re: RTK reports token savings, but our cost benchmarks disagree
#6Main takeaway:
Average cost per attempt, without → with RTK:
Claude/Fable: $1.72 → $1.64 (~5% cheaper)
DeepSeek: $0.115 → $0.121 (~5% more expensive)
Almost all Claude savings came from a single task.
Excluding it, savings were under 1%.
Re: RTK reports token savings, but our cost benchmarks disagree
#7i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
I don't think they're comparable. RTK just modifies the output of CLI tools to reduce the number of tokens, a Roslyn MCP gives the agent a fundamentally superior way of interacting with a C# codebase.
Re: RTK reports token savings, but our cost benchmarks disagree
#8This makes sense. “Don’t try to penny-pinch your employees” is a lesson most managers learn eventually, and I guess agent-orchestrators will have to learn it too.
[dead]
Re: RTK reports token savings, but our cost benchmarks disagree
#9i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.
I don't think they're comparable. RTK just modifies the output of CLI tools to reduce the number of tokens, a Roslyn MCP gives the agent a fundamentally superior way of interacting with a C# codebase.
Yes. and i find model makes less errors and reasoning the codebase well, especially when you do a large refactor.
Re: RTK reports token savings, but our cost benchmarks disagree
#10[flagged]