> "No one has any idea how to create X now" does not lead to "Imminent creation of X is very unlikely to be true". Right?
I think it does lead to that. How will people create something they have no idea how to create... Perhaps it will somehow emerge accidentally, but that's very unlikely.
I tried to make this clear before, but I'll state it clearly: I think discussing is completely fine. We all have our hobbies. It's convincing people there's a real problem and taking real money to solve a fake problem that I object to. That's a con.
If AGI does happen and they start doing things we GIs don't like, the problem is going to be around specifically what they are doing, why they are doing it and what we don't like about it. The solutions are going to be in terms of those specific problem spaces. There would also be a feedback loop of response, counter-response, which is fundamentally unpredictable.
> evolution-like processes
Given a planet-sized environment and 4-5 billion years, I think we can say emergent GI is possible. But that doesn't help someone with a lab. I think we'd do better by studying how brains work, really. I think we're probably going to do best by starting with simpler brains and see if we can trace the path evolution has take (thus saving ourself billions of years of random trial and error).
> That's a probability estimate!!
Let me wrap up on this: I estimate a 99.99% likelihood that people discussing the problems of AGI now will fail to correctly predict the nature of AGI and their resultant problems, and therefore will waste any and all effort/money spent now in that direction.
Someone actually seriously interested in this who wants to put resources to it now would want to put money into a long-term trust, to be used once the problem is clear. However, that assumes the existence of a stable trustee mechanism that remains uncorrupted over the long term to ensure the money is used as intended. I'm not how that is accomplished, but some long-term foundations seem to have done OK. (E.g., the Nobel Foundation doesn't do too bad. The Peace Prize is all politics, and all of the prizes have some politics, but overall it's mostly on-track.)
If there's a 10% chance such a trust function well enough long enough for 50% of the money to be spent in search of solutions to the actual problems, that's 5% effectiveness. Far better than the 0.01% effectiveness I estimate for spending the money now.
Note: the ancestor post of this discussion also estimated a 0.01% effectivness:
> If AGI has even a 1% chance of happening this century and those donations reduce the risk of it going wrong and killing everyone by 1% that's still money very well spent.
Still, I think an EA could easily come up with another use for the same resources that does a lot better than 5%, so we're back to not spending any resources on this now. Much better to have future EAs work on it, once the problems are clear.