John Carmack's new AGI company, Keen Technologies, has raised a $20M round

Post by 93po » Sat, Aug 27, 2022, 2:10 PM UTC

trention wrote: post #560

93po wrote: post #498

Earlier quoted context omitted.

> AGI will be more dangerous that nuclear weapons. This is highly debatable and frankly no one on the planet is qualified to know this for sure. It's just as likely it won't be dangerous.

There is enough work already done in game theory about the dangers of nuclear escalation. Current state of affairs pretty much ensures no rational agent will want to use them. On the other hand, when you model the interactions of significantly more intellectually capable rational agents with us, the "calculus" is not very favorable, to put it mildly.

Post by Tepix » Wed, Aug 31, 2022, 9:08 AM UTC

polishdude20 wrote: post #516

HL33tibCe7 wrote: post #182

Earlier quoted context omitted.

It sounds like you haven’t really thought through AI safety in any real detail at all. Airgapping and the necessity of human input are absolutely not ways to prevent an AGI gaining access to a system. A true, superintelligent AGI could easily extort (or persuade) those humans. If you think concerns over AGI are “stupid”, you haven’t thought about it enough. It’s a massive display of ignorance. The Computerphile AI sa…

What I don't understand is what motivates this world destroying AGI? Like, it's got motives right? How does it get them? Do we program them in? If we do, is the fear that it won't stop at anything in its way to fulfill its objective? If so, what stops it from removing that objective from its motivation? If it can discover zero days to fulfill its objective, wouldn't it reason about itself and it's human-set motives a…