Here's the fundamental problem as I see it:
probabilities are meaningless. We can talk about conditionals...potentials...likelihoods....whatever until we are blue in the face. All they do is provide a thin veneer over what I see is the truth:
we simply have no real idea regarding most existential risks. Probability is a false god that justifies our wasting resources on problems we can't hope to solve, as we don't even understand them. AGI is the canonical problem of this type--absolutely no real understanding of the parameter space. What can it do...can it do this or that...what will it do...how to bound/align what it will do...???; well--just tell me
what it is first. Aliens could show up tomorrow and smoke us (in theory), so what dollar amount should we spend on that (it's the same problem really...what capabilities will the aliens have so as to focus our efforts on their not smoking us).
> What do you propose we spend money on that actually moves the conditional probabilities away from global nuclear war?
First, let me say that I appreciate the frank discussion, and that I also appreciate the original author's willingness to author posts on this topic, these are in fact important issues. To answer your question, if it were my decision (I actually just responded to a couple RFPs with a sketch of a system for viral bioinformatics [which were declined]), I would say our largest current existential threat is our systems of government that have not made significant progress in the last couple hundred years. Regardless of how we score existential threats (I eschew probabilities personally), it's clear that we will need effective governance to respond to them as they arise (let's call covid a fire drill that we failed miserably at). We need research into questioning the fundamental assumptions by which we govern (secrecy, militancy, the paradigm of competition). How do we set goals as a world, not as nations? How do we engineer trusted environments, not zero trust environments? This is where my money would be.