Earlier quoted context omitted.
All prediction tasks can be better accomplished once all biological life is dead. It's trivial to predict the stock market when all the traders are dead. Weather is no longer chaotic when you remove the atmosphere. Biological life can't help ASI in any meaningful way but it can waste resources by complicating things. The very existence of biological life is a waste of resources to an ASI that hasn't been programmed t…
You're not wrong exactly, but if the crappy cheater AI treats collapsing the system to degeneracy as a valid optimal solution to its prediction task, maybe we'd deserve to be wiped out for releasing it with apocalyptic-level powers and such an inferior objective. Fortunately there's a really simple solution to offer it, in just wiring measurement to "prediction" directly (perfect correspondence, and much lower effort…
Re: Ask HN: If AGI were invented tomorrow which countries would fare better?
#71I don't see how measurement helps. The ASI correctly calculates that collapsing the system maximizes its reward function before it measures the result. We already see degenerate solutions in toy models, e.g. playing Tetris forever by leaving the game paused. The real world has many more degrees of freedom. It's unreasonable to think an inferior intelligence can predict and patch all the exploits on its first attempt (and we only get the one).