31–40 of 79 posts
So in that case, more models could give a better response, which costs more compute.
Earlier quoted context omitted.
Then why does chain of thought work better than asking for short answers?
Because it’s a better prompt. Works better for people too.
More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
could a training model be fed the raw data or source and weights of an llm and create better functioning llms by spotting patterns and things between models? like if you could feed it all the open source models and it could create sub models off of those and maybe even a 2nd Gen 'self' instance to better train on the second set such that maybe it could find ways to get the same results with 5b model as 75b.
The 75B param model simply has more complexity to work with than the 5B model.
In the same sense that: `y = mx + b` is just not as expressive as `y = ax^2 + bx + c`.
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
- If you have a fixed time budget and increase the GPU memory+compute available, you can directly query a bigger model. Raw models are basically giant lookup functions, and without the extra memory+compute, they'll spill to slower layers of your memory hierarchy, e.g., GPU RAM -> CPU RAM -> disk. Likewise, with MoE models, there are multiple concurrent models being queried.
- Most 'good' LLM systems are not just direct model calls, but code-based agent frameworks on top that call code tools, analyze the results, and decide to edit+retry things. For example, if doing code generation, they may decide to run lint analysis & type checking on a generated output, and if issues, ask the LLM to try again. In Louie.AI, we will even generate database queries and run GPU analytics & visualizations in on-the-fly Python sandboxes. These systems will do backtracking etc retries, and > 50% of the quality can easily come from these layers: LLM leaderboards like HumanEval increasingly report both the raw model + what agent framework on top. All this adds up and can quickly become more expensive than the LLM. So better systems can enable more here too.
Earlier quoted context omitted.
Because it’s a better prompt. Works better for people too.
That's not the only reason. More tokens = more useful compute towards making a prediction. A query with more tokens before the question is literally giving the LLM more "thinking time"
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
You are incorrect. Increasing compute during inference renders similar gains to increasing parameters/compute during training time (see self-consistency, tree of thoughts, etc.)
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
You are incorrect. Increasing compute during inference renders similar gains to increasing parameters/compute during training time (see self-consistency, tree of thoughts, etc.)
I think it has already been implied that we are not talking about increasing the quantity of parameters in this context but the possibily of applying additional compute to a model with a given number of parameters
There's a misconception in the question that is important to address first: when an LLM is running inference it isn't querying its training data at all, it's just using a function that we created previously (the "model") to predict the next word in a block of text. That's it. When considering plain inference (no web search or document lookup), the decisions that determine a model's speed and capabilities come before…
* LLMs are lossy compression functions on their training data.
* The size of the model dictates how lossy the compression is.
* You can't spend compute to get more detail out of a model once it's been compressed/trained, anymore than you can spend compute to get an incredibly lossily-compressed movie to go from 240p back to the original 1080p source.