Live data from Hacker News

Ask HN: Do LLMs get "better" with more processing power and or time per request?

news.ycombinator.com

11–20 of 79 posts

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#11
post #3

More processing power does not make a model better. You can train models on CPUs with same result based on same model architecture and dataset. It'll just take longer to get those results. What makes models "good" is if the dataset "fits" the model architecture properly and you have given it enough time (epochs) to have a semi accurate prediction ratio (lets say 90% accurate). For image classification models I've don…

The OP asks "per request", not training time.

Answer is still no and still for the above reason. Compute resources are only relevant to how fast it can answer not the quality.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#12
The same model will not get better by having more processing power or time. However, that's not the full story.

Larger models generally perform better than smaller models (this is a generalization, but a good enough one for now). The problem is that larger models are also slower.

This ends up being a balancing act for model developers. They could get better results but it may end up being a worse user experience. Models size can also limit where the model can be deployed.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#13

Directly answering your question requires making some assumptions about what you mean and also what "class" of models you are asking about. Unfortunately I don't think it's just a yes or no, since I can answer in both directions depending on the interpretation. [No] If you mean "during inference", then the answer is mostly no in my opinion, but it depends on what you are calling a "LLM" and "processing power", haha.…

They inner layer permutability is super interesting. Is that result published anywhere? That's consistent with this graph here, which seems to imply different layers are kind of working in very related latent spaces. If you skip to the graph here that shows the attention + feed forward displacements tending to align (after a 2d projection), is this something known/understood? Are the attention and feed forward displa…

I haven't published it nor have I seen it published.

I can copy paste some of my raw notes / outputs from poking around with a small model (Phi-1.5) into a gist though: https://gist.github.com/bluecoconut/6a080bd6dce57046a810787f...

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#14
post #8

No, the standard LLM implementations currently used will apply a fixed amount of computations during inference, which is chosen and "baked in" by the model architecture before training. They don't really have the option to "think a bit more" before giving the answer, generating each token makes the exact same amount of matrix multiplications. Well, they probably theoretically could be modified to do it, but we don't…

There's fixed compute per token but more tokens = more compute so a LLM will technically have more "time" for a query with more tokens preceding it.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#15

Directly answering your question requires making some assumptions about what you mean and also what "class" of models you are asking about. Unfortunately I don't think it's just a yes or no, since I can answer in both directions depending on the interpretation. [No] If you mean "during inference", then the answer is mostly no in my opinion, but it depends on what you are calling a "LLM" and "processing power", haha.…

They inner layer permutability is super interesting. Is that result published anywhere? That's consistent with this graph here, which seems to imply different layers are kind of working in very related latent spaces. If you skip to the graph here that shows the attention + feed forward displacements tending to align (after a 2d projection), is this something known/understood? Are the attention and feed forward displa…

Those curves of "embedding displacement" are very interesting!

quickly scanning the blog led to this notebook which shows how they're computed and shows other examples too with similar behavior. https://github.com/spather/transformer-experiments/blob/mast...

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#16
One caveat not mentioned yet is that you can get better responses through priming, fewshot and chain of thought. That means if you start talking about a related problem/concept, mention some keywords, then provide a few examples, then ask the LLM to provide chain of thought reasoning, you will get a better answer. Those will extend the runtime and processing power in practice.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#17
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before the inference step, during the creation of the model.

Building an LLM model consists of defining its "architecture" (an enormous mathematical function that defines the model's shape) and then using a lot of trial and error to guess which "parameters" (constants that we plug in to the function, like 'm' and 'b' in y=mx+b) will be most likely to produce text that resembles the training data.

So, to your question: LLMs tend to perform better the more parameters they have, so larger models will tend to beat smaller models. Larger models also require a lot of processing power and/or time per inferred token, so we do tend to see that better models take more processing power. But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#18
Short answer is No.

I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of).

See: https://youtu.be/zjkBMFhNj_g?t=2100&si=jaImuf3UCn6ReTp4

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#19
There are all sorts of changes one could imagine being made to how LLMs are trained and run, but if you are asking about what actually exists today, then:

1) At runtime, when you feed a "request" (prompt) into the model, the model will use a fixed amount of compute/time to generate each word of output. There is no looping going on internally - just a fixed number of steps to generate each word. Giving it more or less processing power at runtime will not change the output, just how fast that output is generated.

If you, as a user, are willing to take more time (and spend more money) to get a better answer, then a trick that often works is to take the LLM's output and feed it back in as a request, just asking the LLM to refine/reword it. You can do this multiple times.

2) At training time, for a given size of model and given set of training data, there is essentially an optimal amount of time to train for (= amount of computing power and time taken to train). Train for too short a time and the model won't have learnt all that it could. Train for too long a time (repeating the training data), and the model will start to memorize the training set rather than generalize from it, meaning that the model is getting worse.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#20
Interestingly it used to be quite standard with 'small' language models to use a search algorithm to render a full block of text, the most basic being beam search. Then you can get better with more processing power to do a wider path search. This is not what OP is talking about, it just means generating a larger number of candidate continuations. However it's not necessary or optimal for newer LLMs, because it tends to siphon the LLM into quite generic places, and it can get very repetitive.
Post reply on HN