Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Token Compression Illusion: Why I'm Skeptical of RTK

mroczek.dev

Post reply on HN
Print view View original

81–90 of 121 posts

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#81

Post by dfee »

tontinton wrote: post #54
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

an agent harness built in rust with ratatui - checks out. i've built one myself. i don't maintain it, and continue to use opencode, but it was worth it to learn how agent harnesses work.

anyway, what's the real pitch on why i should move on from opencode to maki?

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#83

Post by lackoftactics »

jahala wrote: post #60
lackoftactics wrote: post #30

Author of the text here. I will be honest with why I wrote it, the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs. Now people are wrapping every possible command in rtk and trying to handle all major possible commands and decide which output you should get.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%

Looked at your repo, even starred. On the surface, I like your approach a bit better. It looks like your idea sits at the space between semantic search and compressing tokens. I was into semantic search before, but mostly trying to vectorize codebase instead of tree sitter and couldn’t make the semantic search work for me. Thanks for sharing!

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#84

Post by lackoftactics »

fumeux_fume wrote: post #80
ianwalter wrote: post #46

Earlier quoted context omitted.

Why didn’t you offer any real world usage numbers to illustrate your point? I found this unhelpful.

https://en.wikipedia.org/wiki/Brandolini%27s_law

Thanks for the link! I thought I know every obscure law and I was so wrong.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#85

Post by lackoftactics »

trjordan wrote: post #47

The core of the problem is that there are a million tools that make AI better, and no ways to measure whether AI is working better. Big companies with popular products have it. They do something between normal product analytics and chatbot evals to figure out if users are being successful in their sessions. That's the job. But any given dev, with between 3 and 50 sessions a day? Like, I have no idea what makes the LL…

And the effort to produce valid benchmarks is tremendous. You are probably right and that’s very annoying. We already had flame wars over frameworks and this is way worse, your vibes vs. my vibes. Who would thought non-deterministic outputs would lead us here?

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#86

Post by sdesol »

trjordan wrote: post #47

The core of the problem is that there are a million tools that make AI better, and no ways to measure whether AI is working better. Big companies with popular products have it. They do something between normal product analytics and chatbot evals to figure out if users are being successful in their sessions. That's the job. But any given dev, with between 3 and 50 sessions a day? Like, I have no idea what makes the LL…

> and no ways to measure whether AI is working better.

What I do with my product is I explicity tell you to ask your agent. I have real world examples and real world repositories that you can try with:

https://gitsense.com

https://github.com/gitsense/smart-ripgrep

https://github.com/gitsense/smart-codex

Token saving on average is not what I am mostly interested in though. I am more interested in knowing that the AI doesn't load unnecessary files in context, which can affect reasoning.

You can just ask the agent after a task how many files do you think was not read by knowing the files purpose first?

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#88

Post by jvican »

lackoftactics wrote: post #83
jahala wrote: post #60

Earlier quoted context omitted.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%

Looked at your repo, even starred. On the surface, I like your approach a bit better. It looks like your idea sits at the space between semantic search and compressing tokens. I was into semantic search before, but mostly trying to vectorize codebase instead of tree sitter and couldn’t make the semantic search work for me. Thanks for sharing!

An ex colleague is working on Headroom, a much more legit alternative to RTK. They provide accuracy benchmarks in the repo and are transparent about the compression algorithms used for the different output types. I liked their approach a lot better than RTK and thought it might be relevant for you.

https://github.com/chopratejas/headroom

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#89

Post by jvican »

akman wrote: post #82

Anybody have experience with https://github.com/chopratejas/headroom ? They seem to have similar goals in token reduction, but headroom appears to be broader in scope.

They provide benchmarks as well. I like headroom’s approch a lot better. More transparent.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#90

Post by nullbio »

SubiculumCode wrote: post #3

I feel like what is needed is not compression, but aggressive context management with subagents.

I feel like what is needed is more local tool usage and small local model usage that does the heavy lifting, rather than the paid for LLM burning tokens at all.
Post reply on HN