More processing power does not make a model better. You can train models on CPUs with same result based on same model architecture and dataset. It'll just take longer to get those results. What makes models "good" is if the dataset "fits" the model architecture properly and you have given it enough time (epochs) to have a semi accurate prediction ratio (lets say 90% accurate). For image classification models I've don…
The OP asks "per request", not training time.
Ask HN: Do LLMs get "better" with more processing power and or time per request?
11–20 of 79 posts
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#12Larger models generally perform better than smaller models (this is a generalization, but a good enough one for now). The problem is that larger models are also slower.
This ends up being a balancing act for model developers. They could get better results but it may end up being a worse user experience. Models size can also limit where the model can be deployed.
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#13Directly answering your question requires making some assumptions about what you mean and also what "class" of models you are asking about. Unfortunately I don't think it's just a yes or no, since I can answer in both directions depending on the interpretation. [No] If you mean "during inference", then the answer is mostly no in my opinion, but it depends on what you are calling a "LLM" and "processing power", haha.…
They inner layer permutability is super interesting. Is that result published anywhere? That's consistent with this graph here, which seems to imply different layers are kind of working in very related latent spaces. If you skip to the graph here that shows the attention + feed forward displacements tending to align (after a 2d projection), is this something known/understood? Are the attention and feed forward displa…
I can copy paste some of my raw notes / outputs from poking around with a small model (Phi-1.5) into a gist though: https://gist.github.com/bluecoconut/6a080bd6dce57046a810787f...
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#14No, the standard LLM implementations currently used will apply a fixed amount of computations during inference, which is chosen and "baked in" by the model architecture before training. They don't really have the option to "think a bit more" before giving the answer, generating each token makes the exact same amount of matrix multiplications. Well, they probably theoretically could be modified to do it, but we don't…
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#15Directly answering your question requires making some assumptions about what you mean and also what "class" of models you are asking about. Unfortunately I don't think it's just a yes or no, since I can answer in both directions depending on the interpretation. [No] If you mean "during inference", then the answer is mostly no in my opinion, but it depends on what you are calling a "LLM" and "processing power", haha.…
They inner layer permutability is super interesting. Is that result published anywhere? That's consistent with this graph here, which seems to imply different layers are kind of working in very related latent spaces. If you skip to the graph here that shows the attention + feed forward displacements tending to align (after a 2d projection), is this something known/understood? Are the attention and feed forward displa…
quickly scanning the blog led to this notebook which shows how they're computed and shows other examples too with similar behavior. https://github.com/spather/transformer-experiments/blob/mast...
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#16Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#17Building an LLM model consists of defining its "architecture" (an enormous mathematical function that defines the model's shape) and then using a lot of trial and error to guess which "parameters" (constants that we plug in to the function, like 'm' and 'b' in y=mx+b) will be most likely to produce text that resembles the training data.
So, to your question: LLMs tend to perform better the more parameters they have, so larger models will tend to beat smaller models. Larger models also require a lot of processing power and/or time per inferred token, so we do tend to see that better models take more processing power. But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results.
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#18I highly recommended watching Andrej Karpathy's Intro to LLMs talk, particularly the section on System 1 vs System 2 thinking. Long story short, what you are describing, using more processing to prepare a better response, is something that is an area of interest, but is not currently part of ChatGPT (or any other LLM that I am aware of).
See: https://youtu.be/zjkBMFhNj_g?t=2100&si=jaImuf3UCn6ReTp4
Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?
#191) At runtime, when you feed a "request" (prompt) into the model, the model will use a fixed amount of compute/time to generate each word of output. There is no looping going on internally - just a fixed number of steps to generate each word. Giving it more or less processing power at runtime will not change the output, just how fast that output is generated.
If you, as a user, are willing to take more time (and spend more money) to get a better answer, then a trick that often works is to take the LLM's output and feed it back in as a request, just asking the LLM to refine/reword it. You can do this multiple times.
2) At training time, for a given size of model and given set of training data, there is essentially an optimal amount of time to train for (= amount of computing power and time taken to train). Train for too short a time and the model won't have learnt all that it could. Train for too long a time (repeating the training data), and the model will start to memorize the training set rather than generalize from it, meaning that the model is getting worse.