There's linguistic term for this kind of speech: isolating grammars, which don't decline words and use high context and the bare minimum of words to get the meaning across. Chinese is such a language btw. Don't know what Chinese think about their language being regarded as cavemen language...
Caveman: Why use many token when few token do trick
41–50 of 396 posts
Re: Caveman: Why use many token when few token do trick
#42Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
It's not "units of thinking" its "units of reference"; as long as what it produces references the necessary probabilistic algorithms, itll do just fine.
Re: Caveman: Why use many token when few token do trick
#43Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
I agree with this take in general , but I think we need to be prepared for nuance when thinking about these things. Tokens are how an LLM works things out, but I think it's just as likely as not that LLMs (like people) are capable of overthinking things to the point of coming to a wrong answer when their "gut" response would have been better. I do not content that this is the default mode, but that it is both possibl…
The solution to that is turning off thinking mode or reducing thinking budget.
Re: Caveman: Why use many token when few token do trick
#44Earlier quoted context omitted.
That was my first thought too -- instead of talk like a caveman you could turn off reasoning, with probably better results. Additionally, LLMs do not actually operate in text; much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. So unless the LLM was trained otherwise, making it talk like a caveman is more than just theoretically turning it into a caveman.
> much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. What do you mean by that? It’s literally text prediction, isn’t it?
There was a paper recently that demonstrated that you can input different human languages and the middle layers of the model end up operating on the same probabilistic vectors. It's just the encoding/decoding layers that appear to do the language management.
So the conclusion was that these middle layers have their own language and it's converting the text into this language and this decoding it. It explains why sometime the models switch to chinese when they have a lot of chinese language inputs, etc.
Re: Caveman: Why use many token when few token do trick
#45I think this could be very useful not when we talk to the agent, but when the agents talk back to us. Usually, they generate so much text that it becomes impossible to follow through. If we receive short, focused messages, the interaction will be much more efficient. This should be true for all conversational agents, not only coding agents.
> Usually, they generate so much text that it becomes impossible to follow through. Quite often on reddit I'll write two paragraphs and get told "I'm not reading all that". Really? Has basic reading become a Herculean task?
Not specifically about your case, but some people are usually just more verbose than others and tend to say the same thing more than once, or perhaps haven't found a clear way of articulating their thoughts down to fewer words.
Re: Caveman: Why use many token when few token do trick
#46Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
When it comes to LLM you really cannot draw conclusions from first principles like this. Yes, it sounds reasonable. And things in reality aren't always reasonable. Benchmark or nothing.
There have been papers about introducing thinking tokens in intermediary layers that get stripped from the output.
Re: Caveman: Why use many token when few token do trick
#47Idk I try talk like cavemen to claude. Claude seems answer less good. We have more misunderstandings. Feel like sometimes need more words in total to explain previous instructions. Also less context is more damage if typo. Who agrees? Could be just feeling I have. I often ad fluff. Feels like better result from LLM. Me think LLM also get less thinking and less info from own previous replies if talk like caveman.
Fluff adds probable likeness. Probablelikeness brings in more stuff. More stuff can be good. More stuff can poison.
Re: Caveman: Why use many token when few token do trick
#48Earlier quoted context omitted.
> much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. What do you mean by that? It’s literally text prediction, isn’t it?
There was a paper recently that demonstrated that you can input different human languages and the middle layers of the model end up operating on the same probabilistic vectors. It's just the encoding/decoding layers that appear to do the language management. So the conclusion was that these middle layers have their own language and it's converting the text into this language and this decoding it. It explains why some…
Ok — that sounds more like a theory rather than an open-and-shut causal explanation, but I’ll read the paper.
Re: Caveman: Why use many token when few token do trick
#49caveman multilingo? how sound?
Re: Caveman: Why use many token when few token do trick
#50Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
What do you mean? The page explicitly states:
> cutting ~75% of tokens while keeping full technical accuracy.
I have no clue if this claim holds, but alas, just pretending they did not address the obvious criticism, while they did, is at the very least pretty lazy.
An explanation that explains nothing is not very interesting.