Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do LLMs get "better" with more processing power and or time per request?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 79 posts

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#41

Post by Lerc »

rohanmehta1 wrote: post #23
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

> But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results. There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.

And adding some more here. I don't know if any models doing this but there is the possibility of generating tokens that it does not show to the user. I think there's quite a lot of scope for internal monologue/chain of thought that could provide concise but clever answers. The difficulty in this is the latency while it ponders to itself, but having played with the groq demos. I think there's scope for a decent interactive experience.

The concern people might feel when they realise an ai might have private thoughts is another issue entirely.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#42

Post by rafaelero »

Lerc wrote: post #39
rafaelero wrote: post #37

Earlier quoted context omitted.

You are incorrect. Increasing compute during inference renders similar gains to increasing parameters/compute during training time (see self-consistency, tree of thoughts, etc.)

Can you elaborate upon that? Apart from the multiplication and accumulations of activations and weights what additional computations can be applied to improve the outputs. I think it has already been implied that we are not talking about increasing the quantity of parameters in this context but the possibily of applying additional compute to a model with a given number of parameters

You can train a smaller model and run inference multiple times and it will reach similar performance as a larger model running inference just once. What's the best way to make use of those multiple inferences is still up to debate, but we already know it works (self-consistency is one example).

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#43

Post by mewpmewp2 »

janalsncm wrote: post #31

My understanding of GPT4 is that it is a mixture of experts. In other words, multiple GPT 3.5 models responding to the same prompt in parallel, and another model on top choosing the best response among them. So in that case, more models could give a better response, which costs more compute.

Where did you get that understanding? This doesn't really make any sense, how would GPT be able to stream token at a time in the first place?

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#44

Post by famouswaffles »

p1esk wrote: post #36
famouswaffles wrote: post #32

Earlier quoted context omitted.

That's not the only reason. More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"

It’s not clear that more tokens are better.

I think it's pretty clear

https://arxiv.org/abs/2310.02226

I mean, i can imagine you wouldn't always need the extra compute.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#45

Post by p1esk »

famouswaffles wrote: post #44
p1esk wrote: post #36

Earlier quoted context omitted.

It’s not clear that more tokens are better.

I think it's pretty clear https://arxiv.org/abs/2310.02226 I mean, i can imagine you wouldn't always need the extra compute.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no clue if it helps or not.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#46

Post by [deleted] »

p1esk wrote: post #45
famouswaffles wrote: post #44

Earlier quoted context omitted.

I think it's pretty clear https://arxiv.org/abs/2310.02226 I mean, i can imagine you wouldn't always need the extra compute.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

[deleted]

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#47

Post by famouswaffles »

p1esk wrote: post #45
famouswaffles wrote: post #44

Earlier quoted context omitted.

I think it's pretty clear https://arxiv.org/abs/2310.02226 I mean, i can imagine you wouldn't always need the extra compute.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

I just found this paper i read a while ago. Doesn't this answer the question ?

The Impact of Reasoning Step Length on Large Language Models - https://arxiv.org/abs/2401.04925

>They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding.

More tokens is more compute time for the model to utilize, that is completely true.

What they guess is that the model can utilize the extra compute for better predictions even if there's no extra information to accompany this extra "thinking time".

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#48

Post by p1esk »

famouswaffles wrote: post #47
p1esk wrote: post #45

Earlier quoted context omitted.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

I just found this paper i read a while ago. Doesn't this answer the question ? The Impact of Reasoning Step Length on Large Language Models - https://arxiv.org/abs/2401.04925 >They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. More tokens is more compute time for t…

Yes, more tokens means doing more compute, that much is true. The question is whether this extra compute helps or hurts. This question is yet to be answered, as far as I know. I tend to make my GPT-4 questions quite verbose, hoping it helps.

This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#49

Post by famouswaffles »

p1esk wrote: post #48
famouswaffles wrote: post #47

Earlier quoted context omitted.

I just found this paper i read a while ago. Doesn't this answer the question ? The Impact of Reasoning Step Length on Large Language Models - https://arxiv.org/abs/2401.04925 >They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. More tokens is more compute time for t…

Yes, more tokens means doing more compute, that much is true. The question is whether this extra compute helps or hurts. This question is yet to be answered, as far as I know. I tend to make my GPT-4 questions quite verbose, hoping it helps. This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).

>The question is whether this extra compute helps or hurts.

I've linked 2 papers now that show very clearly the extra compute helps. I honestly don't understand what else it is you're looking for.

>This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).

That paper specifically dives in on the effect of the length of the CoT prompt. It makes little sense to say - "oh it's just the better prompt" when Cot prompts with more tokens perform better than the shorter ones even when the shorter ones contain the same information. There is also the clear correlation with task difficulty and length.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#50

Post by gremlinsinc »

spywaregorilla wrote: post #33
gremlinsinc wrote: post #21

Earlier quoted context omitted.

could a training model be fed the raw data or source and weights of an llm and create better functioning llms by spotting patterns and things between models? like if you could feed it all the open source models and it could create sub models off of those and maybe even a 2nd Gen 'self' instance to better train on the second set such that maybe it could find ways to get the same results with 5b model as 75b.

People take a model and continue training it all the time (that is, start with already derived weights of one model and doing more training on it to make it something different). Usually this is done to make the model more purpose fit to a specific task, but it won't often make it generically better assuming the first effort was using the model to its full potential (not "underfit"). The 75B param model simply has mo…

well, i was thinking more like..... something that could spit out an android app because it's source is 5k android apps binary/hex code...i.e. it goes off internals, basically its a model of models. So it could find some common ground between all models, and create a new model that's the best of all of them. Then add itself to that list of models, and start up the next generation to do it all over again, including itself, and keep repeating until it can't get any better maybe, or until it finds a new way of doing training, or something. I guess I'm looking for a way to speedup the ai singularity when ai can build upon itself, or really learn like a human -as in receive new input and it's added to the whole of the thing in real time.
Post reply on HN