Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Caveman: Why use many token when few token do trick

github.com

Post reply on HN
Print view View original

61–70 of 396 posts

Re: Caveman: Why use many token when few token do trick

#61

Post by vova_hn2 »

avaer wrote: post #28
TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

That was my first thought too -- instead of talk like a caveman you could turn off reasoning, with probably better results. Additionally, LLMs do not actually operate in text; much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. So unless the LLM was trained otherwise, making it talk like a caveman is more than just theoretically turning it into a caveman.

> instead of talk like a caveman you could turn off reasoning, with probably better results

This is not how the feature called "reasoning" work in current models.

"reasoning" simply let's the model output and then consume some "thinking" tokens before generating the actual output.

All the "fluff" tokens in the output have absolutely nothing to do with "reasoning".

Re: Caveman: Why use many token when few token do trick

#62

Post by akdor1154 »

samus wrote: post #41

There's linguistic term for this kind of speech: isolating grammars, which don't decline words and use high context and the bare minimum of words to get the meaning across. Chinese is such a language btw. Don't know what Chinese think about their language being regarded as cavemen language...

I thought the term for those were 'sane languages', and I say that as a native English speaker :)

Re: Caveman: Why use many token when few token do trick

#64

Post by vova_hn2 »

pixelpoet wrote: post #19
gozzoo wrote: post #16

I think this could be very useful not when we talk to the agent, but when the agents talk back to us. Usually, they generate so much text that it becomes impossible to follow through. If we receive short, focused messages, the interaction will be much more efficient. This should be true for all conversational agents, not only coding agents.

> Usually, they generate so much text that it becomes impossible to follow through. Quite often on reddit I'll write two paragraphs and get told "I'm not reading all that". Really? Has basic reading become a Herculean task?

> Has basic reading become a Herculean task?

I find LLM slop much harder to read than normal human text.

I can't really explain it, it's just a feeling.

The feeling that it draaaags and draaaaaags and keeeeeps going on and on and on before getting to the point, and by the time I'm done with all the "fluff", I don't care what is the text about anymore, I just want to lay down and rest.

Re: Caveman: Why use many token when few token do trick

#65

Post by lijok »

jaccola wrote: post #52
staminade wrote: post #40

Earlier quoted context omitted.

That’s why you need filler words that contribute little to the sentence meaning but give it a chance to compute/think. This is part of why humans do the same when speaking.

Do you have any evidence at all of this? I know how LLMs are trained and this makes no sense to me. Otherwise you'd just put filler words in every input e.g. instead of: "The square root of 256 is" you'd enter "errr The er square um root errr of 256 errr is" and it would miraculously get better? The model can't differentiate between words you entered and words it generated its self...

You’re conflating training and inference

Re: Caveman: Why use many token when few token do trick

#66

Post by bjackman »

If this really works there would seem to be a lot of alpha in running the expensive model in something like caveman mode, and then "decompressing" into normal mode with a cheap model.

I don't think it would be fundamentally very surprising if something like this works, it seems like the natural extension to tokenisation. It also seems like the natural path towards "neuralese" where tokens no longer need to correspond to units of human language.

Re: Caveman: Why use many token when few token do trick

#67

Post by suddenlybananas »

throw83849494 wrote: post #57
avaer wrote: post #28

Earlier quoted context omitted.

That was my first thought too -- instead of talk like a caveman you could turn off reasoning, with probably better results. Additionally, LLMs do not actually operate in text; much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. So unless the LLM was trained otherwise, making it talk like a caveman is more than just theoretically turning it into a caveman.

You obviously do not speak other languages. Other cultures have different constrains and different grammar. For example thinking in modern US English generates many thoughts, to keep correct speak at right cultural context (there is only one correct way to say People Of Color, and it changes every year, any typo makes it horribly wrong). Some languages are far more expressive and specialized in logical conditions, co…

>Some languages are far more expressive and specialized in logical conditions, conditionals, recursion and reasoning. Like eskimos have 100 words for snow, but for boolean algebra.

This is simply not true.

Re: Caveman: Why use many token when few token do trick

#68

Post by staminade »

jaccola wrote: post #52
staminade wrote: post #40

Earlier quoted context omitted.

That’s why you need filler words that contribute little to the sentence meaning but give it a chance to compute/think. This is part of why humans do the same when speaking.

Do you have any evidence at all of this? I know how LLMs are trained and this makes no sense to me. Otherwise you'd just put filler words in every input e.g. instead of: "The square root of 256 is" you'd enter "errr The er square um root errr of 256 errr is" and it would miraculously get better? The model can't differentiate between words you entered and words it generated its self...

What do you think chain of thought reasoning is doing exactly?

Re: Caveman: Why use many token when few token do trick

#69

Post by otabdeveloper4 »

TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

LLMs don't think at all.

Forcing it to be concise doesn't work because it wasn't trained on token strings that short.

Re: Caveman: Why use many token when few token do trick

#70

Post by vova_hn2 »

ArekDymalski wrote: post #4

While really useful now, I'm afraid that in the long run it might accelerate the language atrophy that is already happening. I still remember that people used to enter full questions in Google and write SMS with capital letters, commas and periods.

> I still remember that people used to enter full questions in Google

I think that, in the early days of internet search, entering full questions actually produced worse results than just a bunch of keywords or short phrases.

So it was a sign of a "noob", rather than a mark of sophistication and literacy.

Post reply on HN