Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Token Compression Illusion: Why I'm Skeptical of RTK

mroczek.dev

Post reply on HN
Print view View original

61–70 of 121 posts

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#61

Post by jahala »

trjordan wrote: post #47

The core of the problem is that there are a million tools that make AI better, and no ways to measure whether AI is working better. Big companies with popular products have it. They do something between normal product analytics and chatbot evals to figure out if users are being successful in their sessions. That's the job. But any given dev, with between 3 and 50 sessions a day? Like, I have no idea what makes the LL…

There is an answer- these tools should benchmark by cost per correct answer - not just tokens saved.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#62

Post by lackoftactics »

tontinton wrote: post #54
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

What is your approach for reducing token usage and is it different than rtk?

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#63

Post by evilduck »

cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

>the best ideas float up to Codex/Claude imo

They only float up if people create things like RTK and other people try them though.

It's fair to sit this one out and let others figure out if it's worth it or not but tools like RTK, Headroom, caveman mode and others do reduce input and output tokens that need to be processed, and for local LLMs that can have measurable speedups. Whether or not that ultimately hurts the resulting output I don't have enough data to say, but I am happy to play with them to find out.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#64

Post by mingqiz »

cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

But Claude especially copy opensource ideas after they are widely used for months

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#65

Post by chatmasta »

evilduck wrote: post #63
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

>the best ideas float up to Codex/Claude imo They only float up if people create things like RTK and other people try them though. It's fair to sit this one out and let others figure out if it's worth it or not but tools like RTK, Headroom, caveman mode and others do reduce input and output tokens that need to be processed, and for local LLMs that can have measurable speedups. Whether or not that ultimately hurts the…

Also the incentives aren’t exactly aligned. Yes, Anthropic et. al want you to have efficient token usage (because you’ll use it more, and because of some competitive pressure). But it’s not their first priority especially when they make more money with more tokens.

If a tool like rtk improves token efficiency, but has some negative impact on quality, should Anthropic integrate it immediately? Where is the line? This kind of decision is arguably better left to the user.

What they should maybe do, is have a parameter similar to effort level, that allows the user to opt into native features for token minimizing. Make the tools available but leave the choice of the fidelity/savings tradeoff up to the user.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#66

Post by tontinton »

rescbr wrote: post #57
tontinton wrote: post #54

Earlier quoted context omitted.

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

Maki is awesome. Thanks! I'm using it on my X220 and it flies in comparison to OpenCode et al.

Enjoy, I can't go back to other agents now, too spoiled by the speed

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#67

Post by tontinton »

lackoftactics wrote: post #62
tontinton wrote: post #54

Earlier quoted context omitted.

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

What is your approach for reducing token usage and is it different than rtk?

The biggest ones are: using tree-sitter to index code files as a tool, code_execution tool running a workflow of tools inside a python interpreter (monty), and not being a harness developed by the company profiting from selling you the shovels (and introducing "dynamic workflows" aka spawning 50 agents).

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#68

Post by pdimitar »

tontinton wrote: post #54
cityofdelusion wrote: post #22

I am glad articles like this are finally starting to get some momentum around what I call the LLM magic box industry. From caveman mode to RTK to semantic search and everything in between. Developers have become magicians that cast spells instead of engineers. It sucks at work especially with everyone so sure that their magic spell is the one for ultimate token savings. My criteria are: if it’s not in a harness it’s…

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

Wait what? I thought Maki is a Pi/OpenCode replacement i.e. just the TUI for whatever you plug in it i.e. API-based Codex / Claude?

In another comment you said "I can't get back to other agents". What gives? Feels like I completely misunderstood what Maki is.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#69

Post by tontinton »

pdimitar wrote: post #68
tontinton wrote: post #54

Earlier quoted context omitted.

Totally wrong, you underestimate the frontier's incompetence in anything other than building LLM models (ehm ehm flickering TUI for a year "written like a game engine"). I ran a bunch of benchmarks and there are proven ways to reduce tokens while achieving the same results (finding the same CVEs / finding the same bugs in CRs, etc...). See https://maki.sh , it's my own little proof.

Wait what? I thought Maki is a Pi/OpenCode replacement i.e. just the TUI for whatever you plug in it i.e. API-based Codex / Claude? In another comment you said "I can't get back to other agents". What gives? Feels like I completely misunderstood what Maki is.

It's a TUI you're right, but it's also a harness.

As much as I hate to admit, T the tools you provide, the descriptions, and prompts, all amount to pretty big changes in experience, even using the same models.

Re: The Token Compression Illusion: Why I'm Skeptical of RTK

#70

Post by pdimitar »

tontinton wrote: post #69
pdimitar wrote: post #68

Earlier quoted context omitted.

Wait what? I thought Maki is a Pi/OpenCode replacement i.e. just the TUI for whatever you plug in it i.e. API-based Codex / Claude? In another comment you said "I can't get back to other agents". What gives? Feels like I completely misunderstood what Maki is.

It's a TUI you're right, but it's also a harness. As much as I hate to admit, T the tools you provide, the descriptions, and prompts, all amount to pretty big changes in experience, even using the same models.

That didn't help very much. What did you mean by "agents" earlier on? The tool/harness or the LLM itself?

Also -- can you make Maki enforce the underlying LLM to use stuff like fd/rg and not always default to find/grep, for example? And stop trying to do bash-isms in a zsh system?

Post reply on HN