Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

81–85 of 85 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#81

Post by nextaccountic »

vrighter wrote: post #22
nextaccountic wrote: post #12

Earlier quoted context omitted.

What if the next-gen models are trained on RTK output as well? Then you will actually have less tokens in the context window, and the model won't become confused (which would require more turns, wasting tokens)

doesn't change the fact that it doesn't do what it claims to now . I just don't care about vague promises and "trust us bro" vibes that tech is sold for nowadays. It claims x, it doesn't deliver x. Maybe it could in the future, or maybe not.

That's indeed not something that RTK can promise. However, if RTK becomes popular enough, it's unavoidable that its output will start seeping into the training data of future models, which will make RTK perform better

So it's a kind of self-fulfilling prophecy

Re: RTK reports token savings, but our cost benchmarks disagree

#82

Post by theshrike79 »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =)

When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.

Re: RTK reports token savings, but our cost benchmarks disagree

#83

Post by monneyboi »

I use treesitter to build outlines of files and directories: https://github.com/resolveworks/trace

Have not benchmarked it, the intent is mostly to save time, not necessarily tokens. I noticed that the models need a lot of toolcalls to ground themselves, and often have trouble with getting an overview.

Re: RTK reports token savings, but our cost benchmarks disagree

#84

Post by 8note »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

skills have a pretty straightforward token win - you remove the decision making from the agent so it does specific things in a row. at some point youre better off inverting control to a script that only occasionally calls the agent for a small decision

Re: RTK reports token savings, but our cost benchmarks disagree

#85

Post by bunderbunder »

bustermellotron wrote: post #78
bunderbunder wrote: post #52

Earlier quoted context omitted.

I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days. But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity. The caveman gimmick makes sense to me as a…

Similar for ponytail, I don’t know if it saves tokens, but there is less output to read (and usually less over engineering). Occasionally I have to push for more complex code, but that is much nicer than constantly asking for simpler code.

That's also a good point. When I'm using caveman (and especially cavekit), I don't have to spend quite so much energy on dealing with it building features I didn't ask for and don't want.
Post reply on HN