Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Caveman: Why use many token when few token do trick

github.com

Post reply on HN
Print view View original

371–380 of 396 posts

Re: Caveman: Why use many token when few token do trick

#371

Post by tatrions »

benjismith wrote: post #291
krackers wrote: post #281

Earlier quoted context omitted.

Outputting "filler" tokens is also basically doesn't require much "thinking" for an LLM, so the "attention budget" can be used to compute something else during the forward passes of producing that token. So besides the additional constraints imposed, you're also removing one of the ways which it thinks. Explicit COT helps mitigates some of this, but if you want to squeeze out every drop of computational budget you ca…

This is true, but I also think the input context isn't the only function of those tokens... As those tokens flow through the QKV transforms, on 96 consecutive layers, they become the canvas where all the activations happen. Even in cases where it's possible to communicate some detail in the absolute minimum number of tokens, I think excess brevity can still limit the intelligence of the agent, because it starves thei…

[flagged]

Re: Caveman: Why use many token when few token do trick

#372

Post by dysoco »

outworlder wrote: post #285
tapoxi wrote: post #266

This is neat but my employer rates my performance based on token consumption; is there one that makes Claude needlessly verbose?

Is this a joke, or are you serious? Do you work for Nvidia?

I know at least of a major LATAM company which has dashboards to see AI usage per employee and they will call your attention if you don't use it enough.

Re: Caveman: Why use many token when few token do trick

#373

Post by dahart »

ArekDymalski wrote: post #4

While really useful now, I'm afraid that in the long run it might accelerate the language atrophy that is already happening. I still remember that people used to enter full questions in Google and write SMS with capital letters, commas and periods.

My kids made fun of me yesterday when they saw me using a question mark in a search query.

Re: Caveman: Why use many token when few token do trick

#374

Post by chappyasel »

tatrions wrote: post #371
benjismith wrote: post #291

Earlier quoted context omitted.

This is true, but I also think the input context isn't the only function of those tokens... As those tokens flow through the QKV transforms, on 96 consecutive layers, they become the canvas where all the activations happen. Even in cases where it's possible to communicate some detail in the absolute minimum number of tokens, I think excess brevity can still limit the intelligence of the agent, because it starves thei…

[flagged]

[dead]

Re: Caveman: Why use many token when few token do trick

#375

Post by qsera »

K0balt wrote: post #322
qsera wrote: post #228

Earlier quoted context omitted.

>internalize the concepts. This gives the impression that it is doing something more than pattern matching. I think this kind of communication where some human attribute is used to name some concept in the LLM domain is causing a lot of damage, and ends up inadvertently blowing up the hype for the AI marketing...

Except I actually mean to infer the concept of adding things from examples. LLMs are amply capable of applying concepts to data that matches patterns not ever expressed in the training data. It’s called inference for a reason. Anthropomorphic descriptions are the most expressive because of the fact that LLMs based on human cultural output mimic human behaviours, intrinsically. Other terminology is not nearly as expre…

>do manage to effectively convey the external effect

But the problem is that this does not inform about the failure mode. So if I am understanding correctly, you are saying that the behavior of LLM, when it works, is like it has internalized the concepts.

But then it does not inform that it can also say stuff that completely contradicts what it said before, there by also contradicting the notion of having "internalized" the concept.

So that will turn out to be a lie.

Re: Caveman: Why use many token when few token do trick

#376

Post by wktmeow »

So this is really weird, I was using OpenClaw with GPT 5.4 via Codex on I think Friday of last week, and I noticed what looked like thinking tokens spilling to the main chat, and it sounded a lot like this trick! Couple of examples of what I was seeing in the output:

"Need resume task. No skill applies clearly. Need maybe memory? prior work yes need memory_search.” "Need maybe script content from history. Search specific.”

Possible that OpenAI has come up with something very similar here?

Edit: looks like not only me, https://github.com/openclaw/openclaw/issues/25592#issuecomme...

Re: Caveman: Why use many token when few token do trick

#377

Post by TeMPOraL »

weird-eye-issue wrote: post #364
TeMPOraL wrote: post #363

Earlier quoted context omitted.

Do you? I have the same knee-jerk reaction, but if you think about for more than 2 seconds, LLMs at this point have, through training, read much more research about LLMs than any human, so actually, it's not a dumb thing to do. It may not be very current, though.

> read much more research about LLMs than any human How long a response is from an LLM is going to be completely individual based on the system prompt and the model itself. You can read all of the "LLM research" in the world and it's not going to give you a correct generalized answer about this topic. It's not like this is some inherent property of LLMs.

FWIW, they also wrote down something that's so obvious you don't have to know much about LLMs to know that it's true. Even the "stochastic parrot" / "glorified Markov chain" / "regurgitation machine" camps people should be on the same page - LLMs are trained on human communication, and in human communications, longer queries, good manners and correct grammar are associated with longer, more correct and quality responses; correctly, shitposting is associated with shitposts in reply.

That much is, again, obvious. My previous comment was addressing your ridiculing the notion of discussing LLMs with LLMs, which was a fair reaction back in GPT-3.5 era, but not so today.

Re: Caveman: Why use many token when few token do trick

#378

Post by TeMPOraL »

reedlaw wrote: post #289
pxc wrote: post #276

Earlier quoted context omitted.

> Are you saying Chinese is more concise than English? Yeah, definitely. It lacks case and verb conjugations, plus whole classes of filler words, and words themselves are on average substantially shorter. If you listen to or read a hyper-literal transliteration of Chinese speech into English (you can find fun videos of this on Chinese social media), it even resembles "caveman speech" for those reasons. If you look at…

Chinese omits articles, verbs aren't conjugated, and individual characters carry more meaning than English letters, but other than those differences I don't have the impression that Chinese communication is inherently more concise. Some forms of official speech are wordy. Writing is denser, but the amount of information conveyed through speech is about the same. There are jokes about ambiguous words or phrases in bot…

I'm going to guess Chinese and English is going to come out about the same, when someone invents the right metric to compare them. I recall reading about a study somewhere that compared speech in multiple languages wrt. amount of information communicated per second, and the reported result was they were all the same, because speakers of more verbose languages (longer words, simpler grammar) unknowingly compensate speaking faster than baseline.

Re: Caveman: Why use many token when few token do trick

#379

Post by TeMPOraL »

xgulfie wrote: post #134
TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

Ah so obviously making the LLM repeat itself three times for every response it will get smarter

Yes, and observe that people do that too. It gives them more time to notice their own confusion and go "but wait, that's not right" on you.
Post reply on HN