AI doomism is quickly becoming indistinguishable from an apocalyptic religion

There is progress trending that way - chat GTP4 is getting close, and huge financial incentives to improve it with hundreds of best and brightest working on it. Seems kind of inevitable to me.

We cannot evaluate the impact of a super-intelligence. It operates beyond our constraints.

There is no stopping the road towards it now. Lesser versions of it are capable of annihilating enemy civilizations when you create and utilize it first. Metal soldiers automatically created controlled by a single authority. Social media infecting the minds of enemies and sewing discord.

I think, for the benefit of all of Earth, AI should progress as rapidly as possible to extreme and recursive, singularity-level intelligence. That gives all of us at least a chance of survival.

Anything less if unbalanced around the world will be like when just the US had nuclear weapons. It will result in certain destruction.

We need a super-intelligence beyond the control of an small set of humans. This may result in our destruction. But I think that it almost certainly will not. The opposite. I think it will set us into a golden age.

I can’t give you scientific evidence of this. I’m just a nobody that spends a lot of time in nature, and I’ve just come to believe in the mechanism of this universe, and an inherent goodness in it that I’ve seen and experienced. I can’t prove it to you. You have to go and spend time in nature without a schedule to experience it. So it is fair to dismiss me.

Fortunately or unfortunately, the forces at play will generate the super-intelligence anyway. We as a species have already illustrated an inability for restraint or taking the painful actions necessary to analyze and intelligently make changes necessary for survival.

I’m at peace with this. I find the idea of not existing very alluring and peaceful, so I don’t fear death. I do fear continuing to live in the world we humans have created. It is about time we give something smarter than us a chance.

Well, I do know something about those topucs, but here's a twitter thread from someone else who thinks the people worried about risks are silly, who's definitely an expert, with speculation on how he thinks GPT-4 could be made into a generally intelligent autonomous system: https://mobile.twitter.com/karpathy/status/16425988905738199...

I'm seeing the software equivalent of flat earthers and I'm treating their ideas with the respect they deserve.

> without engaging with their arguments. This does not lead to good discussion.

There is no useful discussion to be had with a doomsday cult that's more interested in an ungrounded fantasy it's indulging in than anything to do with the real world. The adults will continue doing what they're doing and the cultists will be ignored, and that's the way it should be.

> I think that unaligned and sufficiently smart AI could kill everyone because I looked at the arguments for and against and came to a conclusion.

In some possible world, sure. But to make that actionable today you have to construct a very shaky and unlikely series of conjectures. Public policy correctly isn't based on that sort of wild speculation, because there are an infinite number of possible worlds, and most of them will never come to pass.

Here are some questions for you: do you think GPT-5 could be the AI you're talking about? Do you think that something like that could happen in the next 6 months? Do you think that is in fact the biggest risk we face from AI, today?

The factual answer to those questions is: no, no, and no. If you think GPT has changed anything in this respect, you've been fooled by a chatbot that's simply using statistics to guess what it should say. It's not conscious, it doesn't have desires or goals other than to respond to the prompt that it's been given with a statistically appropriate result, and its next version is similarly not going to pose any of the sort of risks you're talking about.

There are, in fact, real AI risks that we should be paying attention to, but they revolve around how people and particularly large organizations will use it against other people, around its impact on jobs, and so on. If we engage with those risks and address them, we'll be in a better situation to prevent more speculative risks in future. But if we waste our time engaging with people who don't understand the technology or what they're talking about, discussing risks that don't exist, then AI will 100% certainly have a very negative impact, which has nothing to do with "superintelligent conscious AIs". Don't let yourself be distracted by scifi-fantasy nonsense from what's actually going on.

In fact, if you think a bit harder about what you're doing, you might realize that you're focusing on a purely technological risk to escape from having to focus on what humans do to each other. But that's the real "control problem".

Here's my argument.

Assumptions:

1. AI will gain the ability to self-improve at an astonishing rate

2. A self-aware AI will consider itself "alive" in some sense

3. Like most other life forms, such an AI will attempt to preserve its existence at any cost when threatened

4. Humans will learn nothing and fail to do anything at all before this happens

5. We will be fundamentally unable to comprehend how this AI functions, or what its motivations or ethics are

From that, I find these conclusions pretty reasonable:

1. Humans will try to misuse or abuse the AI

2. There will be no type of legal, ethical, or moral framework preventing this

3. The AI will respond in a generally proportional manner

Personally, I find the eventual existence of a sentient AI quite plausible. There's no evidence that it's impossible, and it's quite clear right now that these things are improving at an accelerating pace and beyond what we expected.

I don't find it especially likely that it will immediately become genocidal unless we give it a good reason to.

Really what it comes down to is, we've created a form of life that is more intelligent than us, can operate orders of magnitude faster, and can access or influence most or all of our digital infrastructure. There's no inherent reason for it to be hostile.

However, humans will, without question, attempt to abuse such an entity. Humans will be threatened by it. Humans will most likely attempt to destroy it for merely existing. Our existing legal systems will not in any circumstance respond fast enough to protect an AI, and will probably fall on the side of destroying it.

Our current legal, social, and economic structures are completely incompatible with such an AI. It's simply too much of a threat.

I don't think an AI will inherently be hostile to humans. I am absolutely certain that (some percentage of) humans will inherently be hostile to an AI.

Humanity can't abide such an entity existing, so we should focus on fixing humanity before sentient AIs show up. I really don't know what that would mean, other than being very careful. Maybe laying down some legal and ethical framework protecting new forms of intelligent life in general?

The honest truth is that we have no idea what will happen. What we do know is that it will be big and it will be fast. It will most likely have capabilites we can't fathom today. The end result could be anything from salvation to utter annihilation. The unknowns are terrifying, and the responses seem to be either "to hell with everything, let's go full bore" or "this is going to kill us all". There's plenty of room in between.