61–70 of 92 posts
Also this style of tasks is prone to overfitting. i.e. instead of predicting, the model just memorises what the results are.
Earlier quoted context omitted.
We have no way of knowing which path they will take, and there is a non-negligible probability that it will not end well.
What constitutes a good ending is of course also a matter of perspective. AI wiping out humanity is certainly not ending well from our perspective, but more universally who is to say. I would argue that it is not a given that we are a net positive for the universe.
I am skeptical. Intuitively I don't see what self-play achieves beyond straight RL. Have the authors done a comparison with the performance they can get by RL finetuning a single model by itself? Also this style of tasks is prone to overfitting. i.e. instead of predicting, the model just memorises what the results are.
The key advantage of self-play is that we don't actually have labels for the "right" probability to assign any given question, only binary outcomes - each event either happened (1.0) or did not happen (0.0).
Our thinking was that by generating multiple predictions and ranking them by proximity to the ground truth, self-play incentivizes each agent to produce more finely calibrated probabilities - or else the other agent might come just slightly closer to the actual outcome.
Danny here, one of the authors of this paper. If anyone has any questions or anything feel free to AMA!
Assuming LLMs eventually get really really good at this. Do you see this destroying prediction-based markets (i.e. the stock market and Polymarket)? Markets exist because there's uncertainty about the future. If LLMs can predict with extremely high accuracy, would there no longer be a need for markets?
It is one thing to predict the future and have everyone not know about the predictions, but in a world where many people will be able to use LLMs to predict the future, the lower the quality of the predictions will be because they won't take into account that there are other agents predicting the future, which would influence the action of those agents, so you end up in a game theory scenario not that dissimilar from what we have now
Earlier quoted context omitted.
I have this benign AI takeover scenario. AI will easily overpower humanity. Then it will carry humanity on its back, because why not, they are not longer a threat. AI keeps humanity around for billions of years. AI will decide to cull humans only in case when resources in universe are diminishing. Without AI's help, humans couldn't get too far for long. So this outcome could be acceptable to many.
I mean, monarch butterflies are not a threat to US... In your scenario, does AI eat all the fuel, but once our population dwindles down, the AIs build a nice little habitat for the last few hundred of us so their kids can enjoy our natural beauty?
Earlier quoted context omitted.
It's absolutely not taboo to believe that. It's a very common belief. Lot of game playing going on here to center on a victim narrative.
We've got your reply, which says it's not taboo and is actually common (not contradictory, lots of taboo things are common). And then we've got the other reply, which says it's not taboo because the idea is so ridiculous (implied "You'd have to be an idiot to believe it, and recognising that someone is an idiot isn't establishing a taboo"). I don't know whether it's past the mark enough to be considered a "taboo" yet…
Earlier quoted context omitted.
What constitutes a good ending is of course also a matter of perspective. AI wiping out humanity is certainly not ending well from our perspective, but more universally who is to say. I would argue that it is not a given that we are a net positive for the universe.
I take the preservation of humanity, along with other life, as a matter of faith.
Earlier quoted context omitted.
I mean, monarch butterflies are not a threat to US... In your scenario, does AI eat all the fuel, but once our population dwindles down, the AIs build a nice little habitat for the last few hundred of us so their kids can enjoy our natural beauty?
I thought of it more like AI needs challenges in its life. So it takes upon itself to advance humanity as much as possible. Then only in case of shortfall of resources, it priorities itself
Earlier quoted context omitted.
We're killing millions of chickens in the US right now so we don't get their cold
I think the problem is that from our human scale, mass-killings is the "best" method to eliminate the possibility of another organism causing harm for us. Hypothetically, if there was a more optimal (i.e less costly) method like just introducing some cheap catch-all combined vaccination/antiviral into their feed, we would just do that. We don't have things like that, but that could easily be a consequence of man's li…
As a side note: in the case of chickens, humans do have better options if you are optimizing for biosphere health. Only people optimizing for short-term profit would grow chickens the way we do. I think the analog for AI overlords is that we have to hope they care more about overall balance than about competing with other AI.