My thermometer for prediction models is the day they can predict the weather so there is never any unknown about the forcast. Is when I'll begin to believe its hot out when they tell me.
LLMs can teach themselves to better predict the future
11–20 of 92 posts
Re: LLMs can teach themselves to better predict the future
#12While interesting, the title is obviously a bit misleading.
> Our results on a temporally held-out test set of questions resolving after December 25, 2024 show that for both of the models that we employed our method on, Phi-4 14B [15] and DeepSeek-R1 14B [14], we find accuracy improvements of between 7–10% over the base versions of these models as well as the same models fine-tuned with randomized outcome labels as a control
So 7–10% improvement for small models like DeepSeek-R1-Distill-Qwen-14B and Phi-4-14B, approaching GPT-4o.
It would be interesting if the same holds for DeepSeek-R1-Distill-Qwen-32B which in my experience is far superior to to DeepSeek-R1-Distill-Qwen-14B in almost every way, yet still runnable without DC class GPUs
The Ridge Plots of brier scores is probably a good hint if your application chan benefit based on it's tail dependence?
IMHO this paper is all about making small models work better, and nothing suggests anything about frontier models or LLMs in general.
Re: LLMs can teach themselves to better predict the future
#13Danny here, one of the authors of this paper. If anyone has any questions or anything feel free to AMA!
Assuming LLMs eventually get really really good at this. Do you see this destroying prediction-based markets (i.e. the stock market and Polymarket)? Markets exist because there's uncertainty about the future. If LLMs can predict with extremely high accuracy, would there no longer be a need for markets?
If your oracle can tell me (and everyone else) the prevailing price of copper in 6 months in a manner which accounts for the reflexivity of everyone suddenly learning what will be the precise prevailing price of copper in 6 months, you've got yourself a perfect universe simulator and I'm not sure what the point is of worrying about any hypotheticals (or copper) at that point.
Re: LLMs can teach themselves to better predict the future
#14but is it really reasoning? honest question re the underlying architecture of transformers also, self play seems quite an intuitive approach. There's another interesting paper from deep mind about play
You can call it blorbblorb if it makes you feel better. Reasoning is a social construct which, for many people, is grounded in humanity. Others ground it using other socially transmitted ontologies. We don't usually discuss how people choose to ground their ontological beliefs, but why not? Why did you choose to ground "reasoning" in the way you do? If you didn't choose, why not?
Throwing your hands up in the air like this doesn't help build a constructive case for using the word reasoning. It builds a case that words mean whatever
Re: LLMs can teach themselves to better predict the future
#15Danny here, one of the authors of this paper. If anyone has any questions or anything feel free to AMA!
Assuming LLMs eventually get really really good at this. Do you see this destroying prediction-based markets (i.e. the stock market and Polymarket)? Markets exist because there's uncertainty about the future. If LLMs can predict with extremely high accuracy, would there no longer be a need for markets?
LLMs might get better at making predictions than humans but there are fundamental mathematical laws that limit how accurate they can get. A key result of chaos theory is that many processes take exponentially more work to simulate linearly further into the future, so accurately predicting them far enough in the future quickly grows in hardware requirements to the point where it would take more compute than is available in the known universe. So there's a hard limit on how accurately any phenomena that's a result of chaotic processes (in the mathematical sense) could be predicted in the future.
Re: LLMs can teach themselves to better predict the future
#16Danny here, one of the authors of this paper. If anyone has any questions or anything feel free to AMA!
So did you make money at polymarket with your models? That would be the ultimate proof.
Re: LLMs can teach themselves to better predict the future
#17My thermometer for prediction models is the day they can predict the weather so there is never any unknown about the forcast. Is when I'll begin to believe its hot out when they tell me.
At least you won’t be moving your goalposts anytime soon, if ever
Re: LLMs can teach themselves to better predict the future
#18Earlier quoted context omitted.
You can call it blorbblorb if it makes you feel better. Reasoning is a social construct which, for many people, is grounded in humanity. Others ground it using other socially transmitted ontologies. We don't usually discuss how people choose to ground their ontological beliefs, but why not? Why did you choose to ground "reasoning" in the way you do? If you didn't choose, why not?
Throwing your hands up in the air like this doesn't help build a constructive case for using the word reasoning. It builds a case that words mean whatever
Yes, words mean whatever. See Saussure and Wittgenstein. To advance the claim that words are objective is to confuse the symbolic with the real.
This is generally regarded by engineer-types as false, but societal taboos and power structures can be revealed by noting what speech provokes the strongest reactions.
Re: LLMs can teach themselves to better predict the future
#19Re: LLMs can teach themselves to better predict the future
#20Earlier quoted context omitted.
Throwing your hands up in the air like this doesn't help build a constructive case for using the word reasoning. It builds a case that words mean whatever
Yes, words mean whatever. See Saussure and Wittgenstein. To advance the claim that words are objective is to confuse the symbolic with the real. This is generally regarded by engineer-types as false, but societal taboos and power structures can be revealed by noting what speech provokes the strongest reactions.
"societal taboos and power structures can be revealed by noting what speech provokes the strongest reactions"
Ok I'll bite. Who is the marginalized Other?