Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Token Compression Illusion: Why I'm Skeptical of RTK

mroczek.dev

Post reply on HN
Print view View original

111–120 of 121 posts

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#111

Post by Zababa »

jahala wrote: post #60
lackoftactics wrote: post #30

Author of the text here. I will be honest with why I wrote it, the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs. Now people are wrapping every possible command in rtk and trying to handle all major possible commands and decide which output you should get.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%

That's already better than RTK because you measure task accuracy AND savings! So I'm more confident in this one than in the RTK/caveman/ponytail stuff.

There are still two things that bother me:

1) I don't really know when tilth is called, how it works kind of. Does the model itself select it when it needs it? Do you need to instruct the model to use it?

2) If the model itself chooses to use it, I'd like to have a benchmark of non regressions on tasks where tilth isn't helping, to ensure you made the model + harness + tools as a whole better rather than more specialized ; or be upfront about more specialized and have more details when to use/when to not use.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#112

Post by Zababa »

compuficial wrote: post #17

> 1. Gamified Savings vs. Your Actual API Bill Tool use output represents a large amount of my output. I'll take 3.7M tokens saved on 3.9M tokens of input. Tokens saved are tokens saved. > 3. Where Are the Accuracy Benchmarks? As a user of RTK, it would be nice to see accuracy benchmarks. However, I've seen no evidence of the model missing anything critical as a result of the compression. As part of their design phil…

I don't think being very strict about preserving correctness is enough. Considering the cost differences between the latest model and an open weight one that's behind, or between the biggest model and the one below it, I think you have to measure performance very carefully.

Rather than the criticism needing to be backed up with evidence, it's up to RTK to prove they don't degrade performance.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#113

Post by 0123456789ABCDE »

jbellis wrote: post #72

I feel bad that I wasted my time reading this. On the points in the article: 1. Yes, "gain" is a vanity metric but it's harmless, nobody is being "fooled" here. 2. This could be a problem in principle, sure, but unless you're actually vetting bug reports you're just spreading FUD. 3. Again, do you have any reason to believe that the thousands of devs using rtk are silently tanking their performance without noticing?…

hope you feel better knowing your effort, reading and then commenting, is appreciated here, and convinced me to read OP's article. it's short, and raises valid points, but i'm left wondering why your reply is so defensive

let me try that style

  1. it's not *just vanity* if it feeds into *rtk*'s pitch. it's the hook, it's meant to convince users, *rtk* will reduce token waste.
  2. OP's article is not spreading fear, uncertainty, or doubt. at best it disputes *rtk*'s claims that it is effective in reducing token waste, and it does so directly with the question: "Where Are the Accuracy Benchmarks?"
  3. a) *beep* - you are disqualified for failing to identify the *burden of proof* obligation lies with *rtk*, not OP; b) OP made no claims, except for the ones you conveniently dismiss — the github issues. furthermore the "reason[s] to believe that the thousands of devs using rtk are silently tanking their performance without noticing" was already answered. you missed it because you couldn't see past the joy of having your pull-request recently merged.
  4. really, you were so disturbed by the article, you couldn't even ignore the *one* non-technical point, in an article *you choose* to interpret as being technical — all of it being your own fault. nevermind how relevant it is as a signal for the effectiveness of such technics.
  5. is it inherent? are we doomed to live with broken tool outputs? note, the issue, here, is not that *rtk* will fail when output changes, *that* is inherent to *rtk*'s current implementation — as i understand it, but that "it will fail quietly, feeding corrupted or partial text to your agent".
you are not better informed, than gp, because you have commits to your name in rtk. you're just biased by the proximity. we're all at a loss for how effective rtk is, because there are no benchmarks measuring its performance beyond some "vanity metric[s]".

you were so close to getting it here:

> instead of reporting that SOMEONE SHOULD MEASURE THIS, you could, you know, measure it yourself

but hey, thanks for getting me to take another look at rtk & co., i am now further convinced these are just the flavor of the month tricks for speed running context rot

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#115

Post by jahala »

Zababa wrote: post #111
jahala wrote: post #60

Earlier quoted context omitted.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%

That's already better than RTK because you measure task accuracy AND savings! So I'm more confident in this one than in the RTK/caveman/ponytail stuff. There are still two things that bother me: 1) I don't really know when tilth is called, how it works kind of. Does the model itself select it when it needs it? Do you need to instruct the model to use it? 2) If the model itself chooses to use it, I'd like to have a be…

[flagged]

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#116

Post by oxavier »

jvican wrote: post #88
lackoftactics wrote: post #83

Earlier quoted context omitted.

Looked at your repo, even starred. On the surface, I like your approach a bit better. It looks like your idea sits at the space between semantic search and compressing tokens. I was into semantic search before, but mostly trying to vectorize codebase instead of tree sitter and couldn’t make the semantic search work for me. Thanks for sharing!

An ex colleague is working on Headroom, a much more legit alternative to RTK. They provide accuracy benchmarks in the repo and are transparent about the compression algorithms used for the different output types. I liked their approach a lot better than RTK and thought it might be relevant for you. https://github.com/chopratejas/headroom

Headroom uses RTK under the hood.

I applaud the benchmarking though.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#117

Post by [deleted] »

celrod wrote: post #101
arcanemachiner wrote: post #29

Earlier quoted context omitted.

The idea itself is sound: If you can reduce the signal-to-noise ratio in the context window, then that's a good thing. Whether or not RTK actually does this has not been established. I would be glad to see some proper benchmarks done on the actual difference this tool makes (not some meaningless "up to 90%" type of language).

I found this, which has some: https://arxiv.org/pdf/2605.28876 TLDR: RTK does not look good according to the author's benchmark.

[deleted]

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#118

Post by jahala »

polski-g wrote: post #94
jahala wrote: post #60

Earlier quoted context omitted.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%

Very cool. I'll probably switch away from AFT to this. Can you add tree-sitter-bash?

Its on the way! https://github.com/jahala/tilth/pull/151

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#119

Post by citizenpaul »

Its interesting that you posted this now. I stopped using RTK about 2 weeks ago due to suspicions and some testing that it may actually be hurting my token usage due to it causing increased loops due to faulty responses to the LLM that confuse it. I only have some rough metrics, unfortunately speed of LLM work has derailed my attempt to nail down usage efficiency. I spend less than $200 per month on tokens anyway and my usage is not consistent. I won't really know for another 30 days when I look at the total billing per day.

So far my token use has not increased.

I also looked through the huge backlog of the RTK issues and got nervous.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#120

Post by citizenpaul »

lackoftactics wrote: post #75
danr4 wrote: post #55

Earlier quoted context omitted.

this is aboslutely entirely written by AI

As an author of the text, I can say you are „absolutely” not correct. I might be already spending too much time with llms and they start to shape my texts, so I am not proud of that either. But thanks for bringing very valuable insight to otherwise interesting discussion.

> they start to shape my texts

I'm a bit curious about this? What draws you to speaking like an LLM rather than your own voice? I've personally never felt compelled to be more llm like in my writing.

Post reply on HN