Machine intelligence, part 2

In addition to what you said, there is a further problem with this line of reasoning: and then what? Doing meaningful intervention on such a system would pretty much require the impossible—solving the halting problem in general. This is something that even the SMI itself wouldn't be able to do because, well, it' impossible.

> I really think a lot of CS people suffer from a kind of Dunning-Krueger effect when it comes to biological systems.

Basically, this. It's not just that we require a huge jump in technology (maybe, maybe not, but probably yes), it's that most of these speculations are incredibly naive. This is literally advising global policy based on the Terminator plot.

About the “conspiracy nut” center of brain. If there is some unstated agenda here, it's hardly a conspiracy any more than the actual system we live in is a conspiracy. Think about it, if we get an SMI which would genuinely work to help humanity, what is the most likely thing to happen? Being a super-intelligent, rational entity it wouldn't take long to figure out that most problems facing humanity is outright irrationality and inefficiency in state organized society and, down votes notwithstanding, capitalism. I don't think I need to convince many here of all the problems with state bureaucracies. As for capitalism, let's just consider, by now somewhat famous essay, The Meditations on Moloch, which lays down some pretty chilling game theoretical arguments against our current global economic system. I don't necessarily agree with all it has to say (although I am against capitalism, just to be honest about political biases here), but what it says sounds very similar to what an intelligent machine would likely work out should it start pondering our problems and possible solutions. If SMI would reach similar conclusions, it seems to me the most rational thing to do would be to just take control away from states and market entities. No need to harm anyone or anything like that, just politely take away their power, just like a parent would take away the knife from a child's hand, and simply, rationally, do what's beneficial to human values. To take the example of Las Vegas from the essay further: it could just say—“No, let's not build a pointless, tacky replica of various cities in the middle of the desert, instead, I'll direct the resources into badly needed fusion power research which would get everyone free energy.” So, these are all intelligent people in power, and if there's any hidden agenda, I'd bet it's this—the fear of losing control and power over society and the rest of humanity. But it's no conspiracy if that's the case, it's just a structurally forced position to take if you're in power (political and/or economic).

But I doubt it too. I agree that is just plain old lack of imagination and too much shallow blockbuster science fiction (BTW, I think the Terminator is a great story, but not a particularly reasonable scenario on which to base your predictions of future scientific, technological and societal development).

Based on history, NASA is a prime example. A government-run organization with a solid track record of contribution to humanity as a whole, not to mention accomplishing feats previously thought impossible.

Obviously co-opting an AGI project in similar fashion would almost certainly be immoral. However, it is worth noting that as a result of NRO's requirements, deploying large payloads such as the Hubble Space Telescope became a reality. Apples and oranges, I suppose.

---

>In fact, I'll be so bold to say that the best way for these fears of SMI to come to life is putting the research under tight government control and regulation.

First, in the context of inevitability, it would be preferable to achieve the advent of AGI as the result of careful intent, rather than accidental discovery.

Second, I was not advocating for centralization nor regulation. The private sector should be free to pursue whatever AI research it desires, unencumbered.

What I was advocating for, is a project of massive scale and resources to compete. Despite the drawbacks of government backing, there is no equal in terms of resources and the ability to enforce secrecy or safety protocols.

If we spin up potentially unsafe AGI in the name of research, would you rather have it contained via airgapped servers in some swanky corporate office, or in an underground facility complete with armed guards? No one would accidentally have a cellphone on them in the latter environment.

They aren't making individually cogent thought out and articulated arguments, they are all just saying "LOUD NOISES, this is a scary thing that someone said could be bad, therefore I think it's a scary thing." If they could give some good timelines, facts and steps toward AGI then we could debate that on merits. As it is no one is doing that, the whole thing is an appeal to authority from the beginning.

So yes, the sources for these people need to be inspected further.