Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

21–30 of 85 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#21

i don't know if such hacks works, but in C# if you use roslyn mcp, you save a lot.

I haven't tried it since it was first released but it didn't seem to work at all for me back then.

It was so slow that the roslyn results would be lagged well behind any edits it was making, which would just leave it confused.

Re: RTK reports token savings, but our cost benchmarks disagree

#22
post #2

well yeah.... now you're giving it output it wasn't trained on.

What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)

doesn't change the fact that it doesn't do what it claims to now. I just don't care about vague promises and "trust us bro" vibes that tech is sold for nowadays. It claims x, it doesn't deliver x. Maybe it could in the future, or maybe not.

Re: RTK reports token savings, but our cost benchmarks disagree

#23

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours? Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec. Releases v0.0.42 Latest last month Why should I trust that what you're peddling isn't snake…

Only way to find out is to do some testing yourself i think.

I’m using less tokens with Lumen but I also use a bunch of other tokens hacks/skills; it’s hard to measure the impact exactly but it feels significant

Re: RTK reports token savings, but our cost benchmarks disagree

#24

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

> One of: Claude Code, Cursor, Codex, or OpenCode

What makes it incompatible with Pi, Zed or any other harness?

Re: RTK reports token savings, but our cost benchmarks disagree

#25
Some months ago I was evaluating command output compressors to integrate into Dirac[1] as that seemed like an easy win that would compliment and compound with Dirac's other mechanisms.

I tested rtk among these and it was actually a net negative in both CPU time and accuracy, the latter would throw LLMs way off and make it hard to recover. If you are building a coding agent, I'd hard pass on rtk.

   ~ $ time grep Return * 2> /dev/null | wc -l
   966
   grep Return * 2> /dev/null 0.36s user 0.02s system 98% cpu 0.382 total
   wc -l 0.00s user 0.00s system 1% cpu 0.380 total


   ~ $ time rtk grep Return * 2> /dev/null | wc -l
   260
   rtk grep Return * 2> /dev/null 4.10s user 17.10s system 92% cpu 23.008 total
   wc -l 0.00s user 0.00s system 0% cpu 23.007 total

Much worse CPU consumption, and more importantly, plain wrong result. These kind of results compromise the entire agent performance because the model trusts wrong output. Without the correct results, any hypothetical savings are penny wise pound foolish

So yeah I am still on the lookout for a credible CLI wrapper, do let me know if you have any in mind.

[1] https://dirac.run/

Re: RTK reports token savings, but our cost benchmarks disagree

#26

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

This sounds quite similar to dirac which made a stir a few months ago:

https://github.com/dirac-run/dirac

I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness.

If any person other than the author has time to verify these Lumen benchmark results I'd be curious to hear it. I don't have the time to do it myself at the moment.

Re: RTK reports token savings, but our cost benchmarks disagree

#28
We have been working in this space for the past year. Based on our experience, I no longer trust any claims unless they are backed by benchmark results (yes, benchmarks are painful to run reliably and expensive).

It is possible to reduce token usage. It’s just much harder than the basic approach.

Re: RTK reports token savings, but our cost benchmarks disagree

#29

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.

You would think, except in their own testing they found no benefit. Some measures it was even worse on.

https://blog.jetbrains.com/ai/2026/05/what-happens-when-you-...

Re: RTK reports token savings, but our cost benchmarks disagree

#30

It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…

Even JetBrains is now AI blog-slop, how disappointing.
Post reply on HN