Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do LLMs get "better" with more processing power and or time per request?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 79 posts

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#51

Post by astrange »

cjbprime wrote: post #40
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

An analogy that works without having to explain anything at all about how LLMs actually work (or maybe does explain a lot, depending on how you look at it) could be: * LLMs are lossy compression functions on their training data. * The size of the model dictates how lossy the compression is. * You can't spend compute to get more detail out of a model once it's been compressed/trained, anymore than you can spend comput…

You obviously can do that though; diffusion models produce better (fsvo better) images the more steps you run of them.

Similarly, LLMs can produce better answers if you teach them thinking strategies that remind them to put the available evidence and intermediate steps in their context window. Otherwise they'll tend to hallucinate an answer out of vaguely correct words.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#52

Post by PeterisP »

famouswaffles wrote: post #14
PeterisP wrote: post #8

No, the standard LLM implementations currently used will apply a fixed amount of computations during inference, which is chosen and "baked in" by the model architecture before training. They don't really have the option to "think a bit more" before giving the answer, generating each token makes the exact same amount of matrix multiplications. Well, they probably theoretically could be modified to do it, but we don't…

There's fixed compute per token but more tokens = more compute so a LLM will technically have more "time" for a query with more tokens preceding it.

A key aspect is the information bottleneck enforced by the mechanism as the next "iteration" only gets to access the new token computed and discards all the other information it computed.

So if you want it to spend more "time" in a useful manner without changing the architecture, you have to get it to write down the temporary information in the tokens, as "think step by step" does or alternatively iterative prompts "write a draft for the rough structure" "now rewrite it better with more detail".

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#53

Post by razodactyl »

famouswaffles wrote: post #32
p1esk wrote: post #30

Earlier quoted context omitted.

Because it’s a better prompt. Works better for people too.

That's not the only reason. More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"

It correlates but the intuition is a bit misleading. What's actually happening is that by asking a model to generate more tokens, it increases the amount of information it has learnt to be present in its context block.

It's why "RAG" techniques work, the models learn during training to make use of information in context.

At the core of self-attention is dot product measurement which causes the model to act like a search engine.

It's helpful to think about it in terms of search: the shape of the outputs look like conversation but were actually prompting the model to surface information from the QKV matrices internally.

Does it feel familiar? When we brainstorm we usually chart graphs of related concepts e.g. blueberry -> pie -> apple.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#54

Post by razodactyl »

p1esk wrote: post #45
famouswaffles wrote: post #44

Earlier quoted context omitted.

I think it's pretty clear https://arxiv.org/abs/2310.02226 I mean, i can imagine you wouldn't always need the extra compute.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

I have a theory that the results are actually a side effect of having the information in a different area of the context block.

Models can be sensitive to the location of a needle in the haystack of its input block.

It's why there are models which are great at single turn conversation but can't hold a conversation past that without multi-turn training.

You can even corrupt the outputs by pushing past the number of turns / show the model data in a form it hasn't really seen before.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#55

Post by razodactyl »

mewpmewp2 wrote: post #43
janalsncm wrote: post #31

My understanding of GPT4 is that it is a mixture of experts. In other words, multiple GPT 3.5 models responding to the same prompt in parallel, and another model on top choosing the best response among them. So in that case, more models could give a better response, which costs more compute.

Where did you get that understanding? This doesn't really make any sense, how would GPT be able to stream token at a time in the first place?

There's actually information provided during token generation that act as a level of confidence.

You can definitely stream and choose the highest scoring values amongst a few shots at generating the best next token candidate.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#56

Post by frannyg »

Lerc wrote: post #39
rafaelero wrote: post #37

Earlier quoted context omitted.

You are incorrect. Increasing compute during inference renders similar gains to increasing parameters/compute during training time (see self-consistency, tree of thoughts, etc.)

Can you elaborate upon that? Apart from the multiplication and accumulations of activations and weights what additional computations can be applied to improve the outputs. I think it has already been implied that we are not talking about increasing the quantity of parameters in this context but the possibily of applying additional compute to a model with a given number of parameters

I wasn't able to elaborate on what I mean with "better" when I asked the question but the idea can indeed be summarized with "will an LLM increase quantity and quality of parameters if you give it more processing power and time". Now I know that language models don't do that at all and that the weights of the user request stored in the "frozen" training data is what assembles the return after generating possible output strings, which are selected by pre-prompts like asking for chain of thought and reasoning paths and so on, which in the end, are nothing more than more weights pulling in more specific context. (I'm just thinking out loud here)

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#57

Post by frannyg »

lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

Right on. A total misconception on my part. And your answer was a nice primer before diving in to the rest of the comments. Thanks!

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#58

Post by frannyg »

cjbprime wrote: post #40
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

An analogy that works without having to explain anything at all about how LLMs actually work (or maybe does explain a lot, depending on how you look at it) could be: * LLMs are lossy compression functions on their training data. * The size of the model dictates how lossy the compression is. * You can't spend compute to get more detail out of a model once it's been compressed/trained, anymore than you can spend comput…

> You can't spend compute to get more detail [...]

Upscaling, technically, is a thing without limits, no?

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#59

Post by frannyg »

rohanmehta1 wrote: post #23
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

> But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results. There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.

That was indeed part of what I wondering about.

Larger and smaller, in my beginner mind, was a difference of much recursiveness the design of the model allowed.

- User request implies knowledge about X. - PULLING in weights for X. - Probability of user knowing about Xm and Xz is low (because the training data says Xm and Xz are PhD-level knowledge or something). - Pulling in weights for an ELI5-level explanation of Xm and Xz ...

I thought, an LLM would do this recursive pulling of weights based on the semantics of the user request, which it does, but it doesn't do that "dynamically" based on "recalculated" weights and regenerated combos of tokens, which could happen if the training data wasn't "frozen" and accessible, which I learned further down in the comments, isn't.

That's why I wondered whether more processing power and or time would benefit this recursive generation and pulling.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#60

Post by frannyg »

tracerbulletx wrote: post #34
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

This doesn't change the point of your answer, but to add on, the result of that learned function is the probability of all tokens occurring next which is sampled when inference is happening. The type of sampling used can be different at inference time.

I'm still figuring out "inference time" but what left me puzzled at first was that there is - to humans at least - an infinite amount of tokens that might come next, technical jargon, synonyms, lexical levels in general, so in my mind there was an RNG build into the function, that, after "filtering" the weights based on the user request - and a lot of different tokens, even those meaning the same or almost the same have the same weights - simply rolled the dice to produce the return string.

I thought the LLM was "getting to know the user" but it had it a short memory span (the context) and thus "forgot" already calculated weights that it would use to (re)generate new weights.

Further down I learned it freaking forgets all the previous weights in general (I think that's what I learned, I'm getting there)

Post reply on HN