Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

61–70 of 87 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#61

Post by rvz »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?

Re: RTK reports token savings, but our cost benchmarks disagree

#62

Post by aeneas_ory »

rvz wrote: post #61
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

So we replace all those "hacks" and snake-oil with even more pseudoscientific hacks and snake-oil?

See my other comment on the person that commented the same thing you did!

Re: RTK reports token savings, but our cost benchmarks disagree

#63

Post by cosmojg »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

How does this compare to ripgrep and fzf?

Re: RTK reports token savings, but our cost benchmarks disagree

#64

Post by stephantul »

I think anyone who is even a little bit realistic knows that most technologies overclaim, or evaluate under very favorable conditions.

This is not a good thing of course, but I also feel that acting surprised that this is going on is a little unnecessary.

Having said that: most tools are not helpful

Re: RTK reports token savings, but our cost benchmarks disagree

#65

Post by kriskrunch »

Open question, how does this instruction in agent-rules.md look?

"Cap large/unknown command output: `COMMAND 2>&1 | head -c 4000`. Never stream full logs, tests, or large files."

I use that instead of RTK. Empirically, I found RTK makes my agents run longer to complete similar tasks.

Ponytail and Caveman seem to help somewhat.

Re: RTK reports token savings, but our cost benchmarks disagree

#66

Post by woadwarrior01 »

kriskrunch wrote: post #65

Open question, how does this instruction in agent-rules.md look? "Cap large/unknown command output: `COMMAND 2>&1 | head -c 4000`. Never stream full logs, tests, or large files." I use that instead of RTK. Empirically, I found RTK makes my agents run longer to complete similar tasks. Ponytail and Caveman seem to help somewhat.

[flagged]

Re: RTK reports token savings, but our cost benchmarks disagree

#67

Post by woadwarrior01 »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

TBF, what difference does it make when the very harnesses you're using these "hacks" in are themselves vibe-coded?

Re: RTK reports token savings, but our cost benchmarks disagree

#68

Post by simianwords »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

> Whether it's caveman

the kinda guy who honestly thinks caveman.md reduces costs, actually adds it to his system prompt and is painstakingly reading the terse output

Re: RTK reports token savings, but our cost benchmarks disagree

#69

Post by gslepak »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

How does lumen compare to semble?

https://github.com/MinishLab/semble

Re: RTK reports token savings, but our cost benchmarks disagree

#70

Post by tracerbulletx »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

Isn't that what everyone was doing in the beginning of harnesses(cursor) and learned not to do for various reasons? https://news.ycombinator.com/item?id=43164253
Post reply on HN