[deleted]
Caveman: Why use many token when few token do trick
181–190 of 396 posts
Re: Caveman: Why use many token when few token do trick
#182Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
CoT token are usually controled via 'extended thinking' or 'adapted thinking'. CoT tokens are usually not affected by the system prompt. There is an effort parameter, though, which states to have an effect on accuracy for over all token consumption. https://platform.claude.com/docs/en/build-with-claude/extend...
This helps, but the original prompt is still there. The system prompt is still influencing these thinking blocks. They just don’t end up clogging up your context. The system prompt sits at the very top of the context hierarchy. Even with isolated "thinking" blocks, the reasoning tokens are still autoregressively conditioned on the system instructions. If the system prompt forces "caveman speak" the model's attention mechanisms are immediately biased toward simpler, less coherent latent spaces. You are handicapping the vocabulary and syntax it uses inside its own thinking process, which directly throttles its ability to execute high-level logic.
Nothing on that page indicates otherwise.
Re: Caveman: Why use many token when few token do trick
#183Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
If this is true, shouldn't LLMs perform way worse when working in Chinese than in English? Seems like an easy thing to study since there are so many Chinese LLMs that can work in both Cbinese and English.
Do LLMs generally perform better in verbose languages than they do in concise ones?
Re: Caveman: Why use many token when few token do trick
#184I disagree with this method and would discourage others from using it too, especially if accuracy, faster responses, and saving money are your priorities.
This only makes sense if you assume that you are the consumer of the response. When compacting, harnesses typically save a copy of the text exchange but strip out the tool calls in between. Because the agent relies on this text history to understand its own past actions, a log full of caveman-style responses leaves it with zero context about the changes it made, and the decisions behind them.
To recover that lost context, the agent will have to execute unnecessary research loops just to resume its task.
Re: Caveman: Why use many token when few token do trick
#185I disagree with this method and would discourage others from using it too, especially if accuracy, faster responses, and saving money are your priorities. This only makes sense if you assume that you are the consumer of the response. When compacting, harnesses typically save a copy of the text exchange but strip out the tool calls in between. Because the agent relies on this text history to understand its own past ac…
me disagree
Re: Caveman: Why use many token when few token do trick
#186I disagree with this method and would discourage others from using it too, especially if accuracy, faster responses, and saving money are your priorities. This only makes sense if you assume that you are the consumer of the response. When compacting, harnesses typically save a copy of the text exchange but strip out the tool calls in between. Because the agent relies on this text history to understand its own past ac…
only you auto-compact. auto-compact bad
Re: Caveman: Why use many token when few token do trick
#187everyone who thinks this is a costly or bad idea is looking past a very salient finding: code doesn't need much language. sure, other things might need lots of language, but code does not. code is already basically language, just a really weird one. we call them programming languages. they're not human languages. they're languages of the machine. condensing the human-language---machine-language interface, good.
if goal make code, few word better. if goal make insight, more word better. depend on task. machine linear, mind not. consider LLM "thinking" is just edge-weights. if can set edge-weights into same setting with fewer tokens, you are winning.
Re: Caveman: Why use many token when few token do trick
#188Earlier quoted context omitted.
> Forcing it to be concise doesn't work because it wasn't trained on token strings that short. This is a 2023-era comment and is incorrect.
LLMs architectures have not changed at all since 2023. > but mmuh latest SOTA from CloudCorp (c)! You don't know how these things work and all you have to go on is marketing copy.
Yea you don't know anything about LLM architectures. They often change with each model release.
You also aren't aware that there's more to it than "LLM architecture". And you're rather confident despite your lack of knowledge.
You're like the old LLMs before ChatGPT was released that were kinda neat, but usually wrong and overconfident about it.
Re: Caveman: Why use many token when few token do trick
#189Earlier quoted context omitted.
LLMs don't think at all. Forcing it to be concise doesn't work because it wasn't trained on token strings that short.
They’re able to solve complex, unstructured problems independently. They can express themselves in every major human language fluently. Sure, they don’t actually have a brain like we do, but they emulate it pretty well. What’s your definition of thinking?
When OP wrote about LLMs "thinking" he implied that they have an internal conceptual self-reflecting state. Which they don't, they *are* merely next token predicting statistical machines.
Re: Caveman: Why use many token when few token do trick
#190Earlier quoted context omitted.
Feel free to elucidate if you want to add anything to this thread other than vibes.
after you go from from millions of params to billions+ models start to get weird (depending on training) just look at any number of interpretability research papers. Anthropic has some good ones.
Interesting, what kind of weird?