I tried this with early ChatGPT. Asked it to answer telegram style with as few tokens as possible. It is also interesting to ask it for jokes in this mode.
It's especially funny to change your coworker's system prompt like that.
Caveman: Why use many token when few token do trick
111–120 of 396 posts
Re: Caveman: Why use many token when few token do trick
#112Earlier quoted context omitted.
LLMs don't think at all. Forcing it to be concise doesn't work because it wasn't trained on token strings that short.
> Forcing it to be concise doesn't work because it wasn't trained on token strings that short. This is a 2023-era comment and is incorrect.
LLMs architectures have not changed at all since 2023.
> but mmuh latest SOTA from CloudCorp (c)!
You don't know how these things work and all you have to go on is marketing copy.
Re: Caveman: Why use many token when few token do trick
#113I didn’t comment on this when I saw it on threads/twitter. But it made it to HN, surprisingly.
I have a feeling these same people will complain “my model is so dumb!”. There’s a reason why Claude had that “you’re absolutely right!” for a while. Or codex’s “you’re right to push on this”.
We’re basically just gaslighting GPUs. That wall of text is kinda needed right now.
Re: Caveman: Why use many token when few token do trick
#114This trick reminds me of "OpenAI charges by the minute, so speed up your audio" https://news.ycombinator.com/item?id=44376989
Which worked great. Also, cut off silences.
> One half interesting / half depressing observation I made is that at my workplace any meeting recording I tried to transcribe in this way had its length reduced to almost 2/3 when cutting off the silence. Makes you think about the efficiency (or lack of it) of holding long(ish) meetings.
Re: Caveman: Why use many token when few token do trick
#115Okay, I like how it reduces token usage, but it kind of feels that, it will reduce the overall model intelligence. LLMs are probabilistic models, and you are basically playing with their priors.
Re: Caveman: Why use many token when few token do trick
#116By the way why don't these LLM interfaces come with a pause button?
And a "prune here" button.
It often happens that the interesting information is in the first paragraph or so, and the remainder is all just the LLM not knowing when to stop. This is super annoying as a conversation then ends up being 90% noise.
Re: Caveman: Why use many token when few token do trick
#117Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
You are absolutely right! That is exactly the reason why more lines of code always produce a better program. Straight on, m8!
Re: Caveman: Why use many token when few token do trick
#118By the way why don't these LLM interfaces come with a pause button?
i imagine they're doing superman level distributed compute across multiple clouds somewhere and cared more about delivering the final result of that than having the ability to pause. which is probably possible, but would require way more work than would be worthwhile. they probably thought the ability to stop and resubmit would be an adequate substitute.
Re: Caveman: Why use many token when few token do trick
#119By the way why don't these LLM interfaces come with a pause button?
i imagine they're doing superman level distributed compute across multiple clouds somewhere and cared more about delivering the final result of that than having the ability to pause. which is probably possible, but would require way more work than would be worthwhile. they probably thought the ability to stop and resubmit would be an adequate substitute.
These models are autoregressive so I doubt they are running them across multiple clouds. And besides, a pause button is useful from a user's pov.
Re: Caveman: Why use many token when few token do trick
#120Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…
[flagged]
Can't you know that tokens are units of thinking just by... like... thinking about how models work?