Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Token Compression Illusion: Why I'm Skeptical of RTK

mroczek.dev

Post reply on HN
Print view View original

51–60 of 121 posts

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#51

Post by lloyd-christmas »

ianwalter wrote: post #46
lackoftactics wrote: post #30

Author of the text here. I will be honest with why I wrote it, the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs. Now people are wrapping every possible command in rtk and trying to handle all major possible commands and decide which output you should get.

Why didn’t you offer any real world usage numbers to illustrate your point? I found this unhelpful.

I read another post oddly similar earlier today that has more explicit data on that authors codebase: https://codepointer.substack.com/p/cutting-llm-token-costs-w...

TLDR; ~3-4% savings to actual API costs with rtk, caveman, and headroom combined, but nothing tangible on if those cost reductions came at a cost of quality. By their calculations, rtk saved them $4.96 on a $926 bill.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#52

Post by Sayrus »

compuficial wrote: post #17

> 1. Gamified Savings vs. Your Actual API Bill Tool use output represents a large amount of my output. I'll take 3.7M tokens saved on 3.9M tokens of input. Tokens saved are tokens saved. > 3. Where Are the Accuracy Benchmarks? As a user of RTK, it would be nice to see accuracy benchmarks. However, I've seen no evidence of the model missing anything critical as a result of the compression. As part of their design phil…

> Tokens saved are tokens saved.

Not always. RTK strips flags and other information. Sometimes you spend more tokens getting them back later. Sure your saved 70% tokens on that tool call, but nothing in the metrics says whether you ran 3 tool calls instead of 1.

There is also a question of whether that stripped output requires more thinking tokens or not.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#53

Post by lackoftactics »

ianwalter wrote: post #46
lackoftactics wrote: post #30

Author of the text here. I will be honest with why I wrote it, the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs. Now people are wrapping every possible command in rtk and trying to handle all major possible commands and decide which output you should get.

Why didn’t you offer any real world usage numbers to illustrate your point? I found this unhelpful.

That’s the fair point. The rtk promotional posts point to 60-90% tokens savings and there is no mention how they perform accuracy wise. The commenter below did great job pointing to resource showing caveman, rtk saving just couple bucks on $926 bill. Thanks, Llyoyd Christmas for linking to useful substack

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#54

Post by tontinton »

cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine").

I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...).

See https://maki.sh, it's my own little proof.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#55

Post by danr4 »

grey-area wrote: post #7
lackoftactics wrote: post #5

Earlier quoted context omitted.

thank you, author here. I will stay civil here and focus on the rtk as that was the goal of article. So do you think rtk cli is ai slop? I had some suspicions looking at their repo and number of issues and their style. The prettier issue with running successfully while binary wasn't even installed was quite entertaining

Did you use an LLM for the blog post? it reads like it in places.

this is aboslutely entirely written by AI

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#56

Post by lopsotronic »

cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Oh, this gold rush has breathed new life into the old school Semantic guys.

Lord knows the DITA priesthood has been running low on rubes, so this new era is a godsend.

Re-coding all of your org's content into a verbose granular schema, that's what will fix these AI things. It's going to give your LLM superpowers! Semantic superpowers!

While everyone completely ignores the utter lack of coupling between the actual language and whatever nonsense is in the element / structure naming. Or the fact that every single thing has to go through some horrible 1990s era parser, which breaks constantly, and now everyone's shovelling the full markup into the very tiny confused mouth of the AI. Or that now everyone needs specialized software to display anything. Or the everything.

My dudes, the thing you're trying to do with this stuff is already done in the vectorizations. You can use math for a lot of it now, instead of someone hand coding "poplar" as "tree" in a totally flat tree structure.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#57

Post by rescbr »

tontinton wrote: post #54
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

Maki is awesome. Thanks! I'm using it on my X220 and it flies in comparison to OpenCode et al.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#58

Post by giancarlostoro »

I just typed in rtk gain on my Mac, unfortunately my main dev machine I reimaged due memory issues I had and it messing up a few things, but on my Mac I've shaved off roughly 51k input tokens, and 23k output tokens, and saved an average of 3 seconds per command. Not sure what the outrage is for or why they cared enough to write this up really.

Not sure who is piping stacktraces through RTK, I only use it for very specific programs, shoving compiler output through it seems silly, but you can always instruct your agent to only use RTK for very specific sets of commands.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#59

Post by cephei »

Many points about maintainability that this article makes seem to hold, especially with update and version output changes, but it doesn't even offer the simplest alternative. Most of these supported commands have flags to strip out noise and reduce output. Maybe agents aren't well trained on these.

As a side note, has anyone tried a dual agent setup where the command output is proxied through a lightweight local model? I can imagine a scenario where all tool output is filtered through Qwen or similar locally to compact the tool output.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#60

Post by jahala »

lackoftactics wrote: post #30

Author of the text here. I will be honest with why I wrote it, the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs. Now people are wrapping every possible command in rtk and trying to handle all major possible commands and decide which output you should get.

Would sincerely love to hear your thoughts on https://www.github.com/jahala/tilth - it’s a different approach than RTK, benchmarked to reduce cost per correct answer by ~40%
Post reply on HN