Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do LLMs get "better" with more processing power and or time per request?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 79 posts

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#61

Post by famouswaffles »

razodactyl wrote: post #53
famouswaffles wrote: post #32

Earlier quoted context omitted.

That's not the only reason. More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"

It correlates but the intuition is a bit misleading. What's actually happening is that by asking a model to generate more tokens, it increases the amount of information it has learnt to be present in its context block. It's why "RAG" techniques work, the models learn during training to make use of information in context. At the core of self-attention is dot product measurement which causes the model to act like a sea…

>What's actually happening is that by asking a model to generate more tokens, it increases the amount of information it has learnt to be present in its context block.

I'm not saying this isn't part of it but even if it's just dummy tokens without any new information, it works.

https://arxiv.org/abs/2310.02226

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#62

Post by frannyg »

rafaelero wrote: post #37
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

You are incorrect. Increasing compute during inference renders similar gains to increasing parameters/compute during training time (see self-consistency, tree of thoughts, etc.)

Yeah, I totally forgot about training time and time of request (aaah, inference time! now I get it.) being completely different points in time because the LLM has no access to the training data anymore.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#63

Post by frannyg »

HarHarVeryFunny wrote: post #19

There are all sorts of changes one could imagine being made to how LLMs are trained and run, but if you are asking about what actually exists today, then: 1) At runtime, when you feed a "request" (prompt) into the model, the model will use a fixed amount of compute/time to generate each word of output. There is no looping going on internally - just a fixed number of steps to generate each word. Giving it more or less…

> there is no looping going on internally

My thoughts after this sentence filled a huge gap I was wondering about, thanks.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#64

Post by frannyg »

fykem wrote: post #3

More processing power does not make a model better. You can train models on CPUs with same result based on same model architecture and dataset. It'll just take longer to get those results. What makes models "good" is if the dataset "fits" the model architecture properly and you have given it enough time (epochs) to have a semi accurate prediction ratio (lets say 90% accurate). For image classification models I've don…

Ok, thanks. My misconception kind of prohibited the insight of a potential (theoretical) assert statement, which is kind of what is meant by

> if the [resulting] dataset "fits" the model architecture properly,

right?

I have too many questions. It seems unreasonable to ask away and I should instead read the studies and some books.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#65

Post by frannyg »

famouswaffles wrote: post #47
p1esk wrote: post #45

Earlier quoted context omitted.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

I just found this paper i read a while ago. Doesn't this answer the question ? The Impact of Reasoning Step Length on Large Language Models - https://arxiv.org/abs/2401.04925 >They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. More tokens is more compute time for t…

So "compute" includes just having more data ... that can also be "ignored"/ "skipped" for whatever reasons (e.g. weights), ok.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#66

Post by frannyg »

PeterisP wrote: post #8

No, the standard LLM implementations currently used will apply a fixed amount of computations during inference, which is chosen and "baked in" by the model architecture before training. They don't really have the option to "think a bit more" before giving the answer, generating each token makes the exact same amount of matrix multiplications. Well, they probably theoretically could be modified to do it, but we don't…

> the same model with give the same result

Is it wrong to think of this as misleading? Don't the results for exactly the same request differ because there are multiple output strings with the same computed weights?

Or do you include "multiple ways to phrase the same" in "same results" and I'm being a noob?

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#67

Post by frannyg »

PeterisP wrote: post #52
famouswaffles wrote: post #14

Earlier quoted context omitted.

There's fixed compute per token but more tokens = more compute so a LLM will technically have more "time" for a query with more tokens preceding it.

A key aspect is the information bottleneck enforced by the mechanism as the next "iteration" only gets to access the new token computed and discards all the other information it computed. So if you want it to spend more "time" in a useful manner without changing the architecture, you have to get it to write down the temporary information in the tokens, as "think step by step" does or alternatively iterative prompts "…

This blew my mind a little as it feels unintuitive to do this since you wouldn't just forget what you based your previous reply on, at least not after some practice with your mind and memory (which I need to catch up on, I must add).

It also feels like a multiplication of required processing power but I have no clue yet how one could use the previous generation of weights of and the tokens themselves to improve, elaborate on, widen the range of predicted potential results.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#68

Post by frannyg »

sweezyjeezy wrote: post #20

Interestingly it used to be quite standard with 'small' language models to use a search algorithm to render a full block of text, the most basic being beam search. Then you can get better with more processing power to do a wider path search. This is not what OP is talking about, it just means generating a larger number of candidate continuations. However it's not necessary or optimal for newer LLMs, because it tends…

Nope, this definitelly fills a few gaps, thanks. I'm still too lazy of thinking about this whole O(n) time thing even though I'm constantly wondering whether "more" or better results could be achieved by throwing CPUs at stuff, hahaha. I rarely think in terms of time in general, just about depth, breadth and clarity.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#69

Post by p1esk »

famouswaffles wrote: post #49
p1esk wrote: post #48

Earlier quoted context omitted.

Yes, more tokens means doing more compute, that much is true. The question is whether this extra compute helps or hurts. This question is yet to be answered, as far as I know. I tend to make my GPT-4 questions quite verbose, hoping it helps. This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood).

>The question is whether this extra compute helps or hurts. I've linked 2 papers now that show very clearly the extra compute helps. I honestly don't understand what else it is you're looking for. >This is completely orthogonal to CoT, which is simply a better prompt - it probably causes some sort of better pattern matching (again very poorly understood). That paper specifically dives in on the effect of the length o…

Yes, the CoT paper does provide some evidence that a more verbose prompt works better. Thank you for pointing me to it.

Though I still don’t quite understand what is going on in the dummy tokens paper - what is “computation width” and why would it provide any benefit?

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#70

Post by p1esk »

razodactyl wrote: post #54
p1esk wrote: post #45

Earlier quoted context omitted.

This paper is a great illustration of how little is understood about this question. They discovered that appending dummy tokens (ignored during both training and inference) improves performance somehow. Don’t confuse their guess as to why this might be happening with actual understanding. But in any case, this phenomenon has little to do with increasing the size of the prompt using meaningful tokens. We still have no…

I have a theory that the results are actually a side effect of having the information in a different area of the context block. Models can be sensitive to the location of a needle in the haystack of its input block. It's why there are models which are great at single turn conversation but can't hold a conversation past that without multi-turn training. You can even corrupt the outputs by pushing past the number of tu…

Models can be sensitive to the location of a needle in the haystack of its input block.

But only if we use some sort of attention optimization. For the quadratic attention algo it shouldn’t matter where the needle is, right?

Post reply on HN