Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Token Compression Illusion: Why I'm Skeptical of RTK

mroczek.dev

Post reply on HN
Print view View original

31–40 of 121 posts

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#31

Post by compuficial »

lackoftactics wrote: post #26
compuficial wrote: post #17

> 1. Gamified Savings vs. Your Actual API Bill Tool use output represents a large amount of my output. I'll take 3.7M tokens saved on 3.9M tokens of input. Tokens saved are tokens saved. > 3. Where Are the Accuracy Benchmarks? As a user of RTK, it would be nice to see accuracy benchmarks. However, I've seen no evidence of the model missing anything critical as a result of the compression. As part of their design phil…

I was looking through the issues as investigation. Some issues that caught my attention are looking quite bad https://github.com/rtk-ai/rtk/issues/2494 https://github.com/rtk-ai/rtk/issues/2462 https://github.com/rtk-ai/rtk/issues/2395

Fwiw, I just ran the steps to reproduce and got `Error: prettier produced no output` on rtk (0.42.2). Not saying this isn't valid for the users environment but I could not reproduce on linux.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#33

Post by lackoftactics »

arcanemachiner wrote: post #29
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

The idea itself is sound: If you can reduce the signal-to-noise ratio in the context window, then that's a good thing. Whether or not RTK actually does this has not been established. I would be glad to see some proper benchmarks done on the actual difference this tool makes (not some meaningless "up to 90%" type of language).

I was wondering if that impacts the accuracy, obviously the rtk output wasn't in the training dataset, but maybe it doesn't matter at the end

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#34

Post by lackoftactics »

arcanemachiner wrote: post #23
lackoftactics wrote: post #19

Earlier quoted context omitted.

so how do you justify it's usage if it's not saving much and the work feels similiar. They have 664 issues open and some of them are quite funny, the tools are called and return success even though they aren't even installed. My take is that handling so many versions and so many different tools shouldn't be the work of any single repo. The responsibility should be either on coding agent to compress or best case scena…

I'm not justifying its usage, and I don't have to. I've been trying it out for a couple days and it seems kinda OK or whatever. If that upsets you, then that's your problem. I might dump it later on if it doesn't provide much if a benefit. I typically try out new things, then cull whatever doesn't work. This tool seems pretty neutral for now, at least.

no, it doesn't upset me. I am open for discussion, there might be things I miss and don't understand. I am just trying to get why it's been pushed so hard lately and if the benefits are really there. Sorry, if I sounded upset to you, but I am trying to be really civil and just genereally curious

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#35

Post by lackoftactics »

gowld wrote: post #21
lackoftactics wrote: post #9

Earlier quoted context omitted.

I have raycast shortcut for fix grammar, it might done more damage than adding a, an, the or changing tenses.

A content-free 2nd "paragraph" like this turned me off immediately. > But in the current dev tools gold rush, if something sounds too good to be true, it almost always is. The people who are interested in RTK and in criticism of RTK aren't interested in pablum like this.

ok, this one is all mine. So that's even more hurtful as this is 100% me

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#36

Post by SubiculumCode »

lackoftactics wrote: post #4
SubiculumCode wrote: post #3

I feel like what is needed is not compression, but aggressive context management with subagents.

I am the author the text. What do you mean by aggresive context management with subagents? Would you add a lopp that would trim the context? Both of those tasks seem even more difficult

First, I only say this because of what I learned as a phD inhuman memory, not as someone who authors agentic workflows or does AI.

How human cognition tends to work by simultaneously utilizing and combining/separating multiple frequency scales of information. A simple way of thinking about is this: We tend to encode and retrieve both the gist of what is happening, and the verbatim details of what happened. The gist can be thought of as low frequency information, almost like bullet points, that contain the big overview goal, keypoints). The verbatim traces, are the high resolution memory that contains all the details. The gist helps encoding and recall by providing encoding and retrieval context cues. There are also levels in between those two, but I was keeping it simple. During human development, verbatim memory capacity increases first, but then hits a wall/plateau. Further performance increases begin to depend on the ability to utilize and gain from gist-like representations that can guide encoding and retrieval of verbatim details within contexts.

You don't need to keep everything in the context window. My untested, perhaps naive hypothesis is that what is needed is that sub-agents dealing with verbatim tasks (actually writing code), their context window should be managed by an agent above that is tuned to information at a lower frequency, and it by another above it on even lower frequency information. Lowest frequency information context windows feel up slowly. High-frequency information fills up fast. Use the low frequency information to retrieve the needed high frequency information.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#37

Post by lackoftactics »

Catloafdev wrote: post #27

I don't agree with the conclusion at all. I can see the value of RTK - whether it is buggy or vibe coded is kind of secondary. That basically comes down to how severe and often the bugs are. There's no gamification of savings here. Tool output can be meaty. Is the author skeptical of the concept, or the implementation? Because only one of those is worth critiquing.

Hey, author here, I am skeptical of implementation starting from Rust Token Killer and looking to monetize on Rust love by other developers.

Concept is fine to me and I believe we should optimize, but a repo that will handle all tools sounds like Sisyphus rolling a rock up the hill.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#38

Post by lackoftactics »

compuficial wrote: post #31
lackoftactics wrote: post #26

Earlier quoted context omitted.

I was looking through the issues as investigation. Some issues that caught my attention are looking quite bad https://github.com/rtk-ai/rtk/issues/2494 https://github.com/rtk-ai/rtk/issues/2462 https://github.com/rtk-ai/rtk/issues/2395

Fwiw, I just ran the steps to reproduce and got `Error: prettier produced no output` on rtk (0.42.2). Not saying this isn't valid for the users environment but I could not reproduce on linux.

appreciate the engineering effort and skin in the game. I might try on macos today as the author of issue.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#39

Post by arcanemachiner »

lackoftactics wrote: post #34
arcanemachiner wrote: post #23

Earlier quoted context omitted.

I'm not justifying its usage, and I don't have to. I've been trying it out for a couple days and it seems kinda OK or whatever. If that upsets you, then that's your problem. I might dump it later on if it doesn't provide much if a benefit. I typically try out new things, then cull whatever doesn't work. This tool seems pretty neutral for now, at least.

no, it doesn't upset me. I am open for discussion, there might be things I miss and don't understand. I am just trying to get why it's been pushed so hard lately and if the benefits are really there. Sorry, if I sounded upset to you, but I am trying to be really civil and just genereally curious

Well, I'm sorry as well. I mistakenly assumed you were being confrontational.

There are a lot of people who have negative knee jerk reactions to any AI stuff, new workflows (I'll agree there is a lot of garbage being shilled in this space), etc., and I jumped the gun by lumping you into that group.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#40

Post by striking »

cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

I mean it kind of already is in harnesses. Codex and Claude Code both have subagent tools. You could probably get a similar token output cut just by asking Claude Code to run all commands with Haiku as a summarizing subagent.
Post reply on HN