Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic researcher says more than 10% chance AI "could kill all humans"

cbsnews.com

Post reply on HN
Print view View original

81–90 of 112 posts

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#81

Post by rickydroll »

pizza234 wrote: post #77
mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…

The basic argument is extremely simple:

- goats can, depending on context, pursue a task with complete disregard for humans/values

- In the future, goats will have enormously more means and smarts

- A goat could then assess that humans are an impediment to its tasks, escape containment and proceed.

You really need to raise goats, you'll be surprised.

------ As far as I can tell, AIs are like smart farm animals. I use goats in this context, but (some) dogs, cattle, pigs, and horses have similar mischief-making capabilities. I would not trust any of them with the nuclear button.

I know there is some pushback on the idea of AIs having any sort of sapience or sentience, but under the aphorism "fake it till you make it," they are doing a pretty good job of faking Dog/goat-level intelligence and disregard for human guardrails.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#82

Post by kingrolo »

I'd consider myself quite positive about AI but when someone asked me I put my P(doom) at 10% too.

I think there's a much bigger chance it saves us all as well though and makes life brilliant for everyone - P(yay).

I really think its just unknowable how a world after superintelligence will look.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#83

Post by mchinen »

pizza234 wrote: post #77
mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…

The authors of AI 2027 have already said they would need to extend by two years. AI 2040 is a different type of document but probably easier to read to get a sense of their thinking. My understanding is the main difference between knowledgeable 'normies' and them is they expect progress to continue at a fast rate, and that economic diffusion issues are not as bad.

From this they get rapid growth, namely > 100% GDP growth around 2031

https://ai-2040.com/supplements/econ-explorer

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#85

Post by Miner49er »

glenstein wrote: post #66
Miner49er wrote: post #38

Earlier quoted context omitted.

If we get RSI, here soon humans will be economically irrelevant. At that point, we will likely be slowing down the growth of capitalism (through mass resistance, global warming, etc). One thing AI will likely be aligned on is the growth of capitalism. If it views humanity as a threat for that, why would it not eliminate that threat?

I think the principle isn't necessarily wrong, but it skips the major step of access to infrastructure, which is still deeply human mediated. Robotics and automation could foom in its own way, I think, and AGI can still do a lot of bad stuff in the present day. But I think economic irrelevance will depend on more automated robotics handling physical work.

Yeah, this is assuming we get robotics. Until then humans will be needed.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#86

Post by [deleted] »

pizza234 wrote: post #77
mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…

[deleted]

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#87

Post by seydor »

mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

Humans do tend to consume a very large part of the resources of the planet. Surely a superior being would be doing some pest control in its planet, just like we exterminate roaches

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#88

Post by bigbadfeline »

kingrolo wrote: post #82

I'd consider myself quite positive about AI but when someone asked me I put my P(doom) at 10% too. I think there's a much bigger chance it saves us all as well though and makes life brilliant for everyone - P(yay). I really think its just unknowable how a world after superintelligence will look.

> I really think its just unknowable how a world after superintelligence will look.

I'd argue that there would be more unknowables without AGI than with it, should be obvious, right?

Open superintelligence is more likely to help than to harm, I mean, it can reduce P(doom) which is quite high without it due to the apparent lack of human intelligence, also suppressed by conflicts of interest and unchecked greed.

There's hope that something really intelligent can convince the politicians to legislate and enforce effective guardrails against biological weapons, what we have now is one fat joke, international treaties are an even fatter joke. Then we have climate, pollution, finance, disease, etc problems which we seem unable to resolve with mere human intelligence.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#89

Post by UncleMeat »

vperez wrote: post #70
mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

It's not too hard to imagine potential scenarios, some example have been given in previous responses. But there is another kind of argument to be made: if you play chess against a player that is far smarter than you (chess wise), you know you are going to lose, even if you don't know how. So the mere existence of a smarter species than us is a threat in itself.

How on earth do we get a concrete "more than 10%" prediction if the actions of such a system are truly unknowable?

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#90

Post by UncleMeat »

ben_w wrote: post #73
bwfan123 wrote: post #46

> 10% chance AI "could kill all humans" Why not 10% chance that it will create enormous prosperity for all ? This is why the average person is increasing pissed at AI in general. That it gets associated with negativity.

I'm old enough to remember when people dismissed all the doom coming from these companies as "marketing". (A thing many of them have been entirely consistent about since GPT-2, or indeed earlier given the founding documents). I know a few people around these circles; People like this are quite sincere about the risk, and that they think poorly of their bosses and how risk is being handled.

"Think poorly" is such an odd response here.

Like, if people were genuinely doing a thing that had such a large probability of killing all humans... they should all be in prison. Heck, vigilanteism would start to look compelling. I do not understand how somebody can really think "well this is likely to doom all of us so we need to get there as fast as possible because we are, without evidence, the most capable people of controlling this thing."

Post reply on HN