Ask HN: Do LLMs get "better" with more processing power and or time per request?

So MoE models are a bit like thinking tools running concurrently, right(?), sieving through training data on paths that are the same contextually, but different in terms of specificity and sensitivity.

If the agents/experts/ architectures - the code - don't have the minimum required amount of memory & processing power, they might even miss entire bunches of tokens that are or might be relevant within the given (the prompt) and predicted/requested context. So more processing power and or time is relevant only to the extent, here: size, of the to-be-queried-at-inference-time training data (tokens and weights).

Now here's where I find myself exactly within the realm that I was in when I phrased my question: analysing the result of a request and evaluating different sets of tokens, which, I now understand, makes much more sense within the subject of code generation than with the recitation of facts or bits of narratives.

Generated code has functions (things to do with other things). Functions can be done more or less efficient, while even the least efficient code works "more than good and fast enough". There is no value in looping through versions of fact and fiction when the answer fits the expectation. And if it doesn't fit, users can have an actual conversation, which is where I get another part of my answer, which is that more processing power only becomes relevant in relation to the amount of concurrent requests in relation to the parts of the training data that are queried at inference time.

No single request will ever query so much data at the same time, that memory and compute become a bottleneck.

It definitely can become a bottleneck when a long/large/broad( but specific) request gets processed by MoEs simultaneously or when versions of results of engineering tasks are being evaluated. But that is simply not within the task or design of current LLMs and is instead added on top (or as a wrapper, for example, which I still fail to find a non-replaceable usecase for while also still being certain that I will find one once I get to LLMs and AIs).

Again, thanks!

Pretty much every Yes and No apply. I had to understand bits of the gaps I was trying to close myself, so thanks for taking the time to interpret into my question.

An LLM can only give probabilities of the next token of output. The time to improve an LLM is during design, training, or fine tuning. Once you've got the final weights, the function is "locked in" and doesn't change.

However part of the process of learning to predict human output from the internet, literature, etc. causes some deeper learning to occur, potentially even more than in humans, certainly of a different nature. The LLM is communicating through a lossy process, and there is some randomness imposed on its outputs, so results may vary.

The nature of the prompt used can trigger some of this deeper learning, and yield better results than you might otherwise get. These weren't put in by design, they are emergent properties of the LLM. For instance "train of thought" prompting has been show to result in better output.

Prompt "engineering" is an empirical process of discovering the quirks and hidden strengths in the model. It is entirely possible that there is a super-human set of cognitive skills embedded inside GPT4, Mistral, or even LLAMA. Given sufficient time, there might be some prompting that could expose it and make it usable.

Because LLMs aren't "programs" in the traditional sense, you should treat them as if they were an alien intelligence, because that is effectively what they are. They don't understand humans, no matter how well they act like it at times. They are wild beasts, and we haven't figured out how to domesticate them yet.

It's viable if you have tools or humans in the loop to comment on them and add new insights.

But the speed isn't really a factor here, and seeing 1000 new apps isn't obviously going to make it better if the model is already at the limits of what it can represent with its parameter count and compression so to speak.