This trick reminds me of "OpenAI charges by the minute, so speed up your audio"
Caveman: Why use many token when few token do trick
31–40 of 396 posts
Re: Caveman: Why use many token when few token do trick
#32Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
This is condescending and wrong at the same time (best combo).
LLMs do stumble into long prediction chains that don’t lead the inference in any useful direction, wasting tokens and compute.
Re: Caveman: Why use many token when few token do trick
#33Cute idea, but you're never gonna blow your token budget on output. Input tokens are the bottleneck, because the agent's ingesting swathes of skills, directory trees, code files, tool outputs, etc. The output is generally a few hundred lines of code and a bit of natural language explanation.
Re: Caveman: Why use many token when few token do trick
#34Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
[deleted]
Re: Caveman: Why use many token when few token do trick
#35[deleted]
Re: Caveman: Why use many token when few token do trick
#36Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
That was my first thought too -- instead of talk like a caveman you could turn off reasoning, with probably better results. Additionally, LLMs do not actually operate in text; much of the thinking happens in a much higher dimensional space that just happens to be decoded as text. So unless the LLM was trained otherwise, making it talk like a caveman is more than just theoretically turning it into a caveman.
> much of the thinking happens in a much higher dimensional space that just happens to be decoded as text.
What do you mean by that? It’s literally text prediction, isn’t it?
Re: Caveman: Why use many token when few token do trick
#37> If caveman save you mass token, mass money — leave mass star.
Mass fun. Starred.
Re: Caveman: Why use many token when few token do trick
#38I think this could be very useful not when we talk to the agent, but when the agents talk back to us. Usually, they generate so much text that it becomes impossible to follow through. If we receive short, focused messages, the interaction will be much more efficient. This should be true for all conversational agents, not only coding agents.
> Usually, they generate so much text that it becomes impossible to follow through. Quite often on reddit I'll write two paragraphs and get told "I'm not reading all that". Really? Has basic reading become a Herculean task?
I think the sentiment here is that the short formulation of Kant's categorical imperative is as good and easier to read than the entirety of "types of ethical theory" (J.J. Martineau).
Re: Caveman: Why use many token when few token do trick
#39Great idea- if the person who made it is reading: Is this based on the board game „poetry for cavemen“? (Explain things using only single-syllable words, comes even with an inflatable log of wood for hitting each other!)
Re: Caveman: Why use many token when few token do trick
#40Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
> Someone didn't get the memo that for LLMs, tokens are units of thinking. Where do you get this memo ? Seems completely wrong to me. More computation does not translate to more "thinking" if you compute the wrong things (ie things that contribute significantly to the final sentence meaning).
That’s why you need filler words that contribute little to the sentence meaning but give it a chance to compute/think. This is part of why humans do the same when speaking.