Live data from Hacker News

AGI Doom and the Drake Equation

iamnotarobot.substack.com

111–113 of 113 posts

Re: AGI Doom and the Drake Equation

#111
post #7

Earlier quoted context omitted.

> The nonexistence of grey goo (von neumann probes) is strong prior for safe agi. what if we're first? or FTL travel isn't possible in our universe?

Sure, have a different prior if you think those assumptions are relevant. i don't P(what if we're first?) ~ 0 P(or FTL travel isn't possible in our universe?) ~ 1

the Grabby Aliens argument is pretty good in establishing why it's likely that we're first-ish. are you familiar with that? or you think it's not convincing? (why?)

Re: AGI Doom and the Drake Equation

#112

As usual with groundbreaking technology, reckless use by military/intelligence agencies are the greatest consequential threat. If we rank unbounded AI on the same potential threat level as nuclear, chemical and biological warfare, than history says that's who will be the first to utilize it in the worst ways possible. I do notice the federal government is going all-in on AI contracts at present, here's the non-black-…

> The fact that nobody's talking about this means it's probably in full swing as we speak

There are people talking about it obviously.

> That's a pretty solid argument for continued public access to these tools, on top of the incredible educational potentials.

We have millions of books yet many people haven't even read the Bible they nominally base their uneducated life on. :|

Re: AGI Doom and the Drake Equation

#113
post #61

Something I do not see represented in these arguments: real world conditions. Most complex computer systems (which we assume to be the case for a super powerful AI) don't run for very long without requiring manual intervention of some sort. "Aha!" I hear you saying, "The AI will figure out how to reboot nodes and scale clusters and such". OK fine. But then there is the meat space... Replacing hardware, running power…

Just as an example of something that's hard, but not because of lack of technology, real economical surplus, or engineering challenges, yet real things which are still killing millions of people each year: malnutrition, access to clean water, and treatable communicable diseases, like TBC or even HIV or COVID.

Solving those are simple tasks of manufacturing power plants, water treatment plants, fertilizer plants, agricultural machines, logistics of deploying them and keeping them fueled, and persuading people to take the fucking vaccine, use PrEP and use condoms. Money solves these. There's enough people and resources to do this.

The AGI foom argument basically says that the AGI will commandeer the economy of the size of a developed country, well, let's say Germany. (Let's pick Germany, because we already saw what that country was able to do when run by a dictator.)

Insisting that AGI can organize enough people to replace enough broken hardware while manufacturing new hardware doesn't seem that far fetched. (Again, Germany was able to increase its industrial production during WWII while the Allies bombed it constantly for months.)

Post reply on HN