> AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us).
I'd say aliens are in the supernova and gamma ray burst territory; even if we understood a lot about them we would be mostly powerless to counter them as an existential threat because there are very few actions we could take to deflect, defeat, or escape alien attack. Reasoning with them is probably the best we could do.
> I would say our largest current existential threat is our systems of government
Just to be clear; do you mean completely wiping out humanity or a reset to some small population that might have another chance at it?
> We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is were my money would be.
This is a lot closer than you may realize to what a lot of AGI existential risk folks research. AGIs will almost certainly be agents in the world; able to act and smart enough to understand their ability to act and affect the world and themselves. The correspondence to governance is pretty close; humans are pretty smart actors who understand a lot of the dynamics they have control over and have goals they want to achieve. AGI will have to fit into an ecosystem of humans and their governments in a beneficial way, and a big part of solving AGI risk is understanding governance, decision theory, goals, and values well enough to not create AGI that turns into a despotic warmonger to achieve its unaligned goals. We know that human agents can end up as despots.
Take the question of how to set goals as the world, not as nations. That extends down to the individual level as well; the coordination problem, unaligned incentives, the tragedy of the commons, decision theory, and other game theory problems can be extended from individuals vs. individuals to nations vs. nations. If anything, AGI will be easier to solve than human governance both because there will be fewer ethical restrictions and because we can (hopefully) engineer AGI in ways that align with human valued and incentives instead of trying to reason with humans.
A trusted environment is, if I take your meaning correctly, one in which nations have aligned their incentives so closely so as to be able to trust that the actions of other nations are honest and in good faith because any other options would be counterproductive. That's at least the sense I get from trust between individuals; after enough time with someone one can be fairly certain that they have enough shared values and goals to be able to come to similar decisions as one would, and trust them to act in accordance either through mutual shared interest (including friendship, admiration, respect, etc.) or the benefit of continued cooperation in the future. I suppose my first thought is that having "nations" to begin with breaks a lot of shared incentives because it overlays unevenly distributed wealth and opportunity with arbitrary legal and social groupings that nevertheless a lot of people care very deeply about. Getting nations to truly cooperate probably means giving up a lot of identity based on tradition, origin, and history. I have no clue how best to convince people that what really matters about them and their neighbors is how individually happy, fulfilled, and free to explore they are regardless of their differences. Getting rid of false beliefs in zero sum economics and the futility of status signalling might be a start, but humans are sort of innately programmed for a lot of counterproductive beliefs about status, pride, fairness, worth, etc.
Not getting too far afield, I am also curious how you reason under uncertainty without probabilities. E.g. you don't know how things may turn out but have some ideas; how do you choose what to do? How will you know if you made a good or bad choice? Is it quantifiable?