Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Do LLMs get "better" with more processing power and or time per request?

news.ycombinator.com

Post reply on HN
Print view View original

21–30 of 79 posts

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#21

Post by gremlinsinc »

lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

could a training model be fed the raw data or source and weights of an llm and create better functioning llms by spotting patterns and things between models? like if you could feed it all the open source models and it could create sub models off of those and maybe even a 2nd Gen 'self' instance to better train on the second set such that maybe it could find ways to get the same results with 5b model as 75b.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#22

Post by teleforce »

lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

This is an excellent concise descriptions on how LLM works, thanks.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#23

Post by rohanmehta1 »

lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

> But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results.

There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#24

Post by pixl97 »

rohanmehta1 wrote: post #23
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

> But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results. There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.

Yea, doing thing like Chain of Thought, and/or running a second query to examine the first set of tokens it generated commonly improve answers.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#25

Post by pixl97 »

chank wrote: post #11
HeavyStorm wrote: post #4

Earlier quoted context omitted.

The OP asks "per request", not training time.

Answer is still no and still for the above reason. Compute resources are only relevant to how fast it can answer not the quality.

Then why does chain of thought work better than asking for short answers?

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#26

Post by lolinder »

rohanmehta1 wrote: post #23
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

> But this is because larger models tend to be better, not because throwing more compute at an existing model helps it produce better results. There's a caveat here - allowing the model to produce more tokens (i.e. giving it more compute time to "think") can produce better results. E.g. asking a model to reason before producing an answer, leads to better answers. And the extra tokens = more compute.

True! It's important to first understand the fundamentals of what makes an LLM "good" and what makes it fast, but yes, there are lots of techniques you can apply right before and during the inference step that can trade off between speed and capabilities.

Different prompting techniques like what you're describing are one way, and RAG [0] and ART [1] are also in a similar category.

[0] https://stackoverflow.blog/2023/10/18/retrieval-augmented-ge...

[1] https://www.promptingguide.ai/techniques/art

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#27

Post by lolinder »

gremlinsinc wrote: post #21
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

could a training model be fed the raw data or source and weights of an llm and create better functioning llms by spotting patterns and things between models? like if you could feed it all the open source models and it could create sub models off of those and maybe even a 2nd Gen 'self' instance to better train on the second set such that maybe it could find ways to get the same results with 5b model as 75b.

I could imagine something like that working in theory, but the amount of examples you would need to train such a model makes it completely impractical. We tend to need billions of examples to get a modern deep learning model working well, and it will be a very long time before reach that many examples of good LLMs.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#29

Post by janalsncm »

gremlinsinc wrote: post #21
lolinder wrote: post #17

There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…

could a training model be fed the raw data or source and weights of an llm and create better functioning llms by spotting patterns and things between models? like if you could feed it all the open source models and it could create sub models off of those and maybe even a 2nd Gen 'self' instance to better train on the second set such that maybe it could find ways to get the same results with 5b model as 75b.

In a way this is already how the model is trained. Model makes a prediction, loss function calculates how “wrong” the prediction was, and we update the weights of the model to minimize the loss.

Re: Ask HN: Do LLMs get "better" with more processing power and or time per request?

#30

Post by p1esk »

pixl97 wrote: post #25
chank wrote: post #11

Earlier quoted context omitted.

Answer is still no and still for the above reason. Compute resources are only relevant to how fast it can answer not the quality.

Then why does chain of thought work better than asking for short answers?

Because it’s a better prompt. Works better for people too.
Post reply on HN