Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

91–99 of 99 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#91

Post by surefirestudios »

bunderbunder wrote: post #52
lopatin wrote: post #50

Earlier quoted context omitted.

I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only…

I am actually rather fond of caveman. I haven't evaluated it for token cost, in part because frankly I think that part of the pitch is a load of malarkey. Output that's shown to the user is such a small percentage of overall tokens these days. But anecdotally I do think it saves me quite a lot of time on reading LLM outputs. And that, if nothing else, is good for my sanity. The caveman gimmick makes sense to me as a…

[flagged]

Re: RTK reports token savings, but our cost benchmarks disagree

#92

Post by hedgehog »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I've gotten good usage out of a structured search tool (not mine, by someone else here) called Tilth:

https://github.com/jahala/tilth

It combines search with tree sitter grammars so the results can annotate usage vs definitions, cite line number ranges, inline the actual definition if it's short, etc. Not as precise as LSP but simple (no daemon), human readable, and in many cases works without configuration.

Re: RTK reports token savings, but our cost benchmarks disagree

#94

Post by patriciobcs »

It seems to depend on the task you are doing. In my case, I use a lot gh and docker CLI, and RTK has save me a lot of usage there. However, using other others CLIs like grep or git doesn’t seem to help that much. I use it mainly for security reviews of codebase and for this particular use case it helps a lot.

Re: RTK reports token savings, but our cost benchmarks disagree

#96

Post by joegibbs »

fg137 wrote: post #35

Glad to see that more and more people realize these are just snake oils. Without objective metrics like benchmarks, none of the claims mean anything. That's also how I feel about skills/plugins. While some provide important context for specific projects/environments, I am very skeptical about (over)generalized skills like "writing JS tests" or "creating a spec". There are dozens of these skills internally at my compa…

Yes I’m also suspicious of skills. I think the main issue is forcing everything down the skill’s path even if it’s not actually necessary.

Re: RTK reports token savings, but our cost benchmarks disagree

#98

Post by wlonkly »

theshrike79 wrote: post #82
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

You use caveman to save on tokens, I use it to make the agent shut up. We are not the same =) When caveman was released, token efficiency was a lot more relevant. But in the time I've actually come to like its matter-of-fact and short way of responding. There are other similar ones, but caveman is familiar.

I was glad to know about caveman when Opus 5 started doing its Opus 5 things.

Re: RTK reports token savings, but our cost benchmarks disagree

#99

Post by master_crab »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

This is the way. Take it with a grain of salt (as the author mentions) but we have gotten much better results and less token usage after we hooked up our codebase to an indexing service’s MCP (in our case sourcegraph). But any indexing service will do.
Post reply on HN