Vecr gives good analogies, but for AGI, I think investing a lot into alignment research, trying to set up incentive structures to actively
avoid "arms races", establishing global oversight and agreements, having a "fire alarm" or process in place that a company can trigger if they develop an AI that they think is close to becoming dangerous, or even individuals. Strong whistleblower protections may help. Tracking GPUs so we know where compute is being gathered together. Starting talks with other countries like China, right now, to build a common understanding of how to deal with a potential problem.
I think it's true that it may be very difficult to solve alignment theoretically, and solving it when we have real-world systems to look at may make it easier (but still probably not easy). But we need to be able to buy companies time so they can slow down as we approach what seems to be the "edge" of dangerous capability / consciousness / goal directedness, without feeling like they will lose out. And to be very strict about the conditions the models are trained under and what testing they undergo before release.
A lot of this is stuff that people are already suggesting, but not everyone is taking the risk seriously.