Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

81–87 of 87 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#81

Post by nextaccountic »

vrighter wrote: post #22
nextaccountic wrote: post #12

Earlier quoted context omitted.

What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)

doesn't change the fact that it doesn't do what it claims to now . I just don't care about vague promises and "trust us bro" vibes that tech is sold for nowadays. It claims x, it doesn't deliver x. Maybe it could in the future, or maybe not.

That's indeed not something that RTK can promise. However, if RTK becomes popular enough, it's unavoidable that its output will start seeping into the training data of future models, which will make RTK perform better

So it's a kind of self-fulfilling prophecy

Re: RTK reports token savings, but our cost benchmarks disagree

#82

Post by theshrike79 »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)

When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.

Re: RTK reports token savings, but our cost benchmarks disagree

#83

Post by monneyboi »

I use treesitter to build outlines of files and directories: https://github.com/resolveworks/trace

Have not benchmarked it, the intent is mostly to save time, not necessarily tokens. I noticed that the models need a lot of toolcalls to ground themselves, and often have trouble with getting an overview.

Re: RTK reports token savings, but our cost benchmarks disagree

#84

Post by 8note »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision

Re: RTK reports token savings, but our cost benchmarks disagree

#85

Post by bunderbunder »

bustermellotron wrote: post #78
bunderbunder wrote: post #52

Earlier quoted context omitted.

I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days. But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity. The caveman gimmick makes sense to me as a…

Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.

That's also a good point. When I'm using caveman (and especially cavekit), I don't have to spend quite so much energy on dealing with it building features I didn't ask for and don't want.

Re: RTK reports token savings, but our cost benchmarks disagree

#86

Post by iharnoor »

Whitespace wrote: post #18
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours? Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec. Releases v0.0.42 Latest last month Why should I trust that what you're peddling isn't snake…

Half of what we see or consume is hype, so we should view everything with nuance. Sometimes truth lies in the middle.

Re: RTK reports token savings, but our cost benchmarks disagree

#87

Post by everforward »

ProjectBarks wrote: post #76
everforward wrote: post #75

Earlier quoted context omitted.

Naively, I think some optimizations would require access to the whole codebase and that would make people nervous (plus incur more cost). Eg absurd idea, but you could write something that minifies a codebase (by token, rather than byte) and then translates edits back into the expanded code. Probably an insane use of fuse lol. Partially minifying on each tool call sounds like a huge pain with a lot of state to track.…

I fully agree. Some of these tautologies, RAGs or RAG adjacent ideas feel like there is real potential. I know in my own testing adding a knowledge graph provided cost saving benefits. Ultimately, the problem is people aren't actually benchmarking any of these ideas or if they are its via an extremely misleading methodology. For example, headroom violates cache frequently, nullifying any potential savings it could ha…

I think people don’t do genuine benchmarks because the market forces them to pretend their solution works for anything you can throw AI at. Companies whose valuation is based on them being the RAG/compression/routing/etc company. They can’t admit it only works well in a specific domain because then they’re immediately $300M in the hole.

I have more faith in companies with a more targeted approach. Eg gzip does fine, but video codecs beat compressing raw video by a ton.

> As an aside, I wonder how many days are we away from Codex or Claude

That sounds like SourceGraph but twice as expensive, although it does have “AI” so probably lol

Post reply on HN