Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Caveman: Why use many token when few token do trick

github.com

Post reply on HN
Print view View original

91–100 of 396 posts

Re: Caveman: Why use many token when few token do trick

#91

Post by jstummbillig »

prodigycorp wrote: post #86
jstummbillig wrote: post #50

Earlier quoted context omitted.

What do you mean? The page explicitly states: > cutting ~75% of tokens while keeping full technical accuracy. I have no clue if this claim holds, but alas, just pretending they did not address the obvious criticism, while they did, is at the very least pretty lazy. An explanation that explains nothing is not very interesting.

The burden of proof is on the author to provide at least one type of eval for making that claim.

I notice that the number of people confidently talking about "burden of proof" and whose it allegedly is in the context of AI has gone up sharply.

Nobody has to proof anything. It can give your claim credibility. If you don't provide any, an opposing claim without proof does not get any better.

Re: Caveman: Why use many token when few token do trick

#92

Post by prodigycorp »

kubb wrote: post #32
TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

This is condescending and wrong at the same time (best combo). LLMs do stumble into long prediction chains that don’t lead the inference in any useful direction, wasting tokens and compute.

Are you sure about that? Chain of thought does not need to be semantically useful to improve LLM performance. https://arxiv.org/abs/2404.15758

Re: Caveman: Why use many token when few token do trick

#93

Post by ShowalkKama »

kogold wrote: post #74
TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

[flagged]

the fact that more tokens = more smart should be expected given cot / thinking / other techniques that increase the model accuracy by using more tokens.

Did you test that ""caveman mode"" has similar performance to the ""normal"" model?

Re: Caveman: Why use many token when few token do trick

#94

Post by davidguetta »

prodigycorp wrote: post #92
kubb wrote: post #32

Earlier quoted context omitted.

This is condescending and wrong at the same time (best combo). LLMs do stumble into long prediction chains that don’t lead the inference in any useful direction, wasting tokens and compute.

Are you sure about that? Chain of thought does not need to be semantically useful to improve LLM performance. https://arxiv.org/abs/2404.15758

still doesn't mean all tokens are useful. it's the point of benchmarks

Re: Caveman: Why use many token when few token do trick

#95

Post by mynegation »

kogold wrote: post #74
TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

[flagged]

No, let me rephrase it for you. “tokens used for think. Short makes model dumb”

Re: Caveman: Why use many token when few token do trick

#96

Post by prodigycorp »

davidguetta wrote: post #94
prodigycorp wrote: post #92

Earlier quoted context omitted.

Are you sure about that? Chain of thought does not need to be semantically useful to improve LLM performance. https://arxiv.org/abs/2404.15758

still doesn't mean all tokens are useful. it's the point of benchmarks

Care to share the benchmarks backing the claims in this repo?

Re: Caveman: Why use many token when few token do trick

#97

Post by zozbot234 »

DimitriBouriez wrote: post #77
nayroclade wrote: post #33

Cute idea, but you're never gonna blow your token budget on output. Input tokens are the bottleneck, because the agent's ingesting swathes of skills, directory trees, code files, tool outputs, etc. The output is generally a few hundred lines of code and a bit of natural language explanation.

Good point and it's actually worse than that : the thinking tokens aren't affected by this at all (the model still reasons normally internally). Only the visible output that gets compressed into caveman... and maybe the model actually need more thinking tokens to figure out how to rephrase its answer into caveman style

Grug says you can tune how much each model thinks. Is not caveman but similar. also thinking is trained with RL so tends to be efficient, less fluffy. Also model (as seen locally) always drafts answer inside thinking then output repeats, change to caveman is not really extra effort.

Re: Caveman: Why use many token when few token do trick

#98

Post by PufPufPuf »

TeMPOraL wrote: post #5

Oh boy. Someone didn't get the memo that for LLMs, tokens are units of thinking . I.e. whatever feat of computation needs to happen to produce results you seek, it needs to fit in the tokens the LLM produces. Being a finite system, there's only so much computation the LLM internal structure can do per token, so the more you force the model to be concise, the more difficult the task becomes for it - worst case, you ca…

You mention thinking tokens as a side note, but their existence invalidates your whole point. Virtually all modern LLMs use thinking tokens.

Re: Caveman: Why use many token when few token do trick

#99

Post by Garlef »

ShowalkKama wrote: post #93
kogold wrote: post #74

Earlier quoted context omitted.

[flagged]

the fact that more tokens = more smart should be expected given cot / thinking / other techniques that increase the model accuracy by using more tokens. Did you test that ""caveman mode"" has similar performance to the ""normal"" model?

Yes but: If the amount is fixed, then the density matters.

A lot of communication is just mentioning the concepts.

Re: Caveman: Why use many token when few token do trick

#100

Post by sillyboi »

Oh, another new trend! I love these home-brewed LLM optimizers. They start with XML, then JSON, then something totally different. The author conveniently ignores the system prompt that works for everything, and the extra inference work. So, it's only worth using if you just like this response style, just my two cents. All the real optimizations happen during model training and in the infrastructure itself.
Post reply on HN