Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

71–80 of 87 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#71

Post by esperent »

nevon wrote: post #48
esperent wrote: post #26

Earlier quoted context omitted.

This sounds quite similar to dirac which made a stir a few months ago: https://github.com/dirac-run/dirac I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness. If any person other than the author has tim…

I did the same with another project that does the same thing, called ck, and wasn't able to wring out any improved performance over just plain grep.

Yes rg + python seems to beat far fancier tools.

Re: RTK reports token savings, but our cost benchmarks disagree

#72

Post by bustermellotron »

Whitespace wrote: post #18
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours? Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec. Releases v0.0.42 Latest last month Why should I trust that what you're peddling isn't snake…

I just went through a lot of benchmarking and the only thing that seemed better than rg was chunkhound, which sounds similar to this project. Actually a small Jina embedding model actually did better than voyage AI, but took a long time to index. Also chunkhound doesn’t work well with worktrees. In the end, I decided to stick with rg.

Re: RTK reports token savings, but our cost benchmarks disagree

#73

Post by chorizo »

elij wrote: post #56

I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.

Surprised this is so far down. A subagent with a cheap model like haiku or similar is the way to go instead of dumping tool output directly into the main agent context

Re: RTK reports token savings, but our cost benchmarks disagree

#74

Post by kgeist »

ericyd wrote: post #46
ProjectBarks wrote: post #17

It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…

> If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money

Judging by the leaks, OpenAI and Anthropic already train reasoning traces to use fewer tokens (they deliberately omit articles and prepositions, use very short sentences, etc.), even though you pay per token. So it wouldn't make sense to do that if the only incentive was "make them pay for as many tokens as possible per task."

It's more subtle than that. If a user has to wait longer for a solution/pay more, they'll be less satisfied and may switch to a competitor. More unnecessary tokens also means more unnecessary compute. Longer sessions are increasingly more expensive to serve than shorter sessions.

And there's always the Jevons effect: as a resource becomes cheaper, demand often increases, and so does net resource consumption.

So, imho, frontier labs have every incentive to reduce token usage per task (while also making you use AI for more and more tasks in your daily life)

Re: RTK reports token savings, but our cost benchmarks disagree

#75

Post by everforward »

ProjectBarks wrote: post #17

It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…

Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost).

Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.

There’s also a lot of common situations where humans prefer solutions that take more tokens because it’s easier for us to read (eg for loop vs map vs list comprehension), which may have some gains.

I strongly suspect there is some form of token compression that works, but I don’t think it will be as simple as “pipe arbitrary text with no context into this tool”.

Jetbrains feels like a place this might come from. “Take this code, parse it to an AST, find the fewest token representation of it” feels like something they’d do, or maybe Astral (specifically in Python land, type checkers feel sort of adjacent as well).

Re: RTK reports token savings, but our cost benchmarks disagree

#76

Post by ProjectBarks »

everforward wrote: post #75
ProjectBarks wrote: post #17

It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…

Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost). Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.…

I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits.

Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could have; but does their benchmarks show that? Nope. [1]

As an aside, I wonder how many days are we away from Codex or Claude taking all the "read" tool calls and re-assembling a full map of your codebase and then offering it as a feature.

[1] https://docs.headroomlabs.ai/docs/benchmarks

Re: RTK reports token savings, but our cost benchmarks disagree

#77

Post by bkotrys »

tracerbulletx wrote: post #70
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253

That’s a good example, I believe it could be similar case like with RAG. I didn’t hear for a while about any recommendations to use RAG.

Re: RTK reports token savings, but our cost benchmarks disagree

#78

Post by bustermellotron »

bunderbunder wrote: post #52
lopatin wrote: post #50

Earlier quoted context omitted.

I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only…

I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days. But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity. The caveman gimmick makes sense to me as a…

Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.

Re: RTK reports token savings, but our cost benchmarks disagree

#79

Post by bustermellotron »

SJMG wrote: post #29
Bridged7756 wrote: post #20

Earlier quoted context omitted.

Jetbrains IDEs are a perfect solution for this. They expose IDE actions (e.g, search, see occurrences, go to implementation) in their MCP server, which the harnesses can then call directly instead of figuring out the code themselves.

You would think, except in their own testing they found no benefit. Some measures it was even worse on. https://blog.jetbrains.com/ai/2026/05/what-happens-when-you-...

I also found LSP like skills to usually have no advantage over rg. The agent needs to read the code to understand it; navigation is a small portion of that.

Re: RTK reports token savings, but our cost benchmarks disagree

#80

Post by bustermellotron »

elij wrote: post #56

I just spawn a subagent in the cheapest range (for example flash-lite) to summarise a tool use. It's the only way that has worked based on my benchmarks and generalises well.

Is there an easy way to do this with eg codex? It seems like eg sol agents can’t spawn Luna subagents, so eg a “code research” subagent can save the main agent’s context, but can’t save tokens necessarily. (I suppose a tool to call codex CLI would work, but a bit unsatisfying.)
Post reply on HN