Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RTK reports token savings, but our cost benchmarks disagree

quesma.com

Post reply on HN
Print view View original

41–50 of 88 posts

Re: RTK reports token savings, but our cost benchmarks disagree

#41

Post by aeneas_ory »

cassianoleal wrote: post #24
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

> One of: Claude Code, Cursor, Codex, or OpenCode What makes it incompatible with Pi, Zed or any other harness?

Only the amount of free time I have to work on it - nothing fundamentally prevents it. PRs welcomed!

Re: RTK reports token savings, but our cost benchmarks disagree

#42

Post by ramon156 »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

Lumen is pretty cool, it's just local RAG, but it's a realistic approach at RAG. I would still keep tool-calling in some places though.

Re: RTK reports token savings, but our cost benchmarks disagree

#43

Post by [deleted] »

Whitespace wrote: post #18
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours? Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec. Releases v0.0.42 Latest last month Why should I trust that what you're peddling isn't snake…

[deleted]

Re: RTK reports token savings, but our cost benchmarks disagree

#45

Post by lucaprata »

aeneas_ory wrote: post #40
Whitespace wrote: post #18

Earlier quoted context omitted.

I should not trust their "vibe-coded productivity/token cost saving hacks" but I should trust yours? Save 30% token costs when using Claude Code, Codex, OpenCode for free - with open source, local semantic search. Works for small and large codebases and monorepos! Enterprise-ready and fully compliant via Ollama and SQLite-vec. Releases v0.0.42 Latest last month Why should I trust that what you're peddling isn't snake…

I literally say you should take benchmarks with a grain of salt :) > Of course, it's always dependent on statistical noise + host system load, and running sufficiently large benchmarks is simply too expensive, so take em with a grain of salt. And the savings listed are coming from a benchmark harness that implements different OSS bugs one time with and one without lumen - in those cases the % saved are reproducible (…

[flagged]

Re: RTK reports token savings, but our cost benchmarks disagree

#46

Post by ericyd »

ProjectBarks wrote: post #17

It seems like most of these tools are mostly vaporware. Benchmarks done on Headroom and RTK show that neither result in real savings. If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves? My guess is they mostly don’t work or make the behavior much more confusing for the model. I really think there needs to be some kind of independent benchmark. Her…

> If it were possible to have such a simple pre-process step why wouldn’t the AI Labs upstream the optimizations themselves?

Not defending these tools, but one reason these might not be upstreamed is because it would negatively impact vendor margins, and they have no incentive to save their users money

Re: RTK reports token savings, but our cost benchmarks disagree

#48

Post by nevon »

esperent wrote: post #26
aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

This sounds quite similar to dirac which made a stir a few months ago: https://github.com/dirac-run/dirac I spent way too long trying to reproduce the results in Pi and failing before I decided that I shouldn't trust author benchmarks for any of these tools. Then I found that I couldn't even close to reproduce their benchmark results using the exact model and their harness. If any person other than the author has tim…

I did the same with another project that does the same thing, called ck, and wasn't able to wring out any improved performance over just plain grep.

Re: RTK reports token savings, but our cost benchmarks disagree

#49

Post by cityofdelusion »

Any magic tool that declares a savings of over 10% can be immediately classified as snake oil. You can check yourself, load any of those projects up in GitHub and notice the math is always extremely misleading. It will be something like theoretical input bytes, or amount of command stripped off, or some other lie.

If the tool won’t be upfront about those things, they are not worth looking into any further. It’s used car salesman strategy.

Re: RTK reports token savings, but our cost benchmarks disagree

#50

Post by lopatin »

aeneas_ory wrote: post #3

All of these "hacks" are snakeoil and I think deep down we all know. Whether it's caveman, RTK, or whatever other vibe-coded productivity/token cost saving hacks/skills/claude.md. What I had success with (although benchmarks are older) is to index the codebase with a dedicated local code embedding model. It's a bit expensive on the CPU side but in my benchmarks it reduced token use and wall clock time significantly.…

I'm in the process of evals for these tools after my org adopted them. My RTK findings are the same. It worsens task performance and overall you don't save money. I wanted to give the same treatment to other tools like ponytail and caveman (especially caveman, I mean there's no way that telling a computer to talk like a caveman is a valid engineering technique right?). To my horror, caveman is looking to be the only tool that actually doesn't regress on reasoning while taking costs down. But I still have a lot more evals to write, so this isn't conclusive or anything. (Also I haven't tried Lumen yet)
Post reply on HN