Caveman: Why use many token when few token do trick • Hacker News Board

Re: Caveman: Why use many token when few token do trick

#101

Post by wzdd » Sun, Apr 05, 2026, 12:22 PM UTC

vova_hn2 wrote: post #59

TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

Yeah, I don't think that "I'd be happy to help you with that" or "Sure, let me take a look at that for you" carries much useful signal that can be used for the next tokens.

They carry information in regular human communication, so I'm genuinely curious why you'd think they would not when an LLM outputs them as part of the process of responding to a message.

Re: Caveman: Why use many token when few token do trick

#102

Post by hybrid_study » Sun, Apr 05, 2026, 12:24 PM UTC

Mongo! No caveman

Re: Caveman: Why use many token when few token do trick

#103

Post by prodigycorp » Sun, Apr 05, 2026, 12:25 PM UTC

jstummbillig wrote: post #91

prodigycorp wrote: post #86

Earlier quoted context omitted.

The burden of proof is on the author to provide at least one type of eval for making that claim.

I notice that the number of people confidently talking about "burden of proof" and whose it allegedly is in the context of AI has gone up sharply. Nobody has to proof anything. It can give your claim credibility. If you don't provide any, an opposing claim without proof does not get any better.

Sorry I don't know how engaging in this could lead to anything productive. There's already literature out there that gives credence to TeMPOraL claim. And, after a certain point, gravity being the reason that things fall becomes so self evident that every re-statements doesnt not require proof.

Re: Caveman: Why use many token when few token do trick

#104

Post by DonHopkins » Sun, Apr 05, 2026, 12:26 PM UTC

Deep digging cave man code reviews are Tha Shiznit:

https://www.youtube.com/watch?v=KYqovHffGE8

Re: Caveman: Why use many token when few token do trick

#105

Post by throw83849494 » Sun, Apr 05, 2026, 12:28 PM UTC

suddenlybananas wrote: post #67

throw83849494 wrote: post #57

Earlier quoted context omitted.

You obviously do not speak other languages. Other cultures have different constrains and different grammar. For example thinking in modern US English generates many thoughts, to keep correct speak at right cultural context (there is only one correct way to say People Of Color, and it changes every year, any typo makes it horribly wrong). Some languages are far more expressive and specialized in logical conditions, co…

>Some languages are far more expressive and specialized in logical conditions, conditionals, recursion and reasoning. Like eskimos have 100 words for snow, but for boolean algebra. This is simply not true.

Well, just take varous english dialects you probably know, there are wast differences. Some strange languages do not even have numbers or recursion.

It is very arrogant to assume, no other language can be more advanced than English.

Re: Caveman: Why use many token when few token do trick

#106

Post by fny » Sun, Apr 05, 2026, 12:29 PM UTC

Are there any good studies or benchmarks about compressed output and performance? I see a lot of arguing in the comments but little evidence.

Re: Caveman: Why use many token when few token do trick

#107

Post by ajd555 » Sun, Apr 05, 2026, 12:32 PM UTC

So, if this does help reduce the cost of tokens, why not go even further and shorten the syntax with specific keywords, symbols and patterns, to reduce the noise and only keep information, almost like...a programming language?

Re: Caveman: Why use many token when few token do trick

#108

Post by adam_patarino » Sun, Apr 05, 2026, 12:33 PM UTC

[flagged]

Re: Caveman: Why use many token when few token do trick

#109

Post by amelius » Sun, Apr 05, 2026, 12:34 PM UTC

By the way why don't these LLM interfaces come with a pause button?

Re: Caveman: Why use many token when few token do trick

#110

Post by DonHopkins » Sun, Apr 05, 2026, 12:34 PM UTC

jerf wrote: post #73

vova_hn2 wrote: post #59

Earlier quoted context omitted.

Yeah, I don't think that "I'd be happy to help you with that" or "Sure, let me take a look at that for you" carries much useful signal that can be used for the next tokens.

There is a study that shows that what the model is doing behind the scenes in those cases is a lot more than just outputting those tokens. For an LLM, tokens are thought. They have no ability to think, by whatever definition of that word you like, without outputting something . The token only represents a tiny fraction of the internal state changes made when a token is output. Clearly there is an optimal for each tas…

High dimensional vectors are thought (insofar as you can define what that even means). Tokens are one dimensional input that navigates the thought, and output that renders the thought. The "thinking" takes place in the high dimension space, not the one dimensional stream of tokens.