Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic researcher says more than 10% chance AI "could kill all humans"

cbsnews.com

Post reply on HN
Print view View original

91–100 of 111 posts

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#91

Post by bigbadfeline »

pizza234 wrote: post #77
mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…

> - AIs can, depending on context, pursue a task with complete disregard for humans/values

Humans can do that way more and way more unhinged than AI, proven too many times by history. There's hoping AI can bring some sense to humans but regardless, the problem isn't AI, it's the natural kind...

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#92

Post by romanows »

ben_w wrote: post #79
romanows wrote: post #34

I'd like to know how these people arrive at their estimates. Why 10% and not 1% or 50%?

Mine is around 10%. There's lots of moving parts and we all have to input our best-guesses as to how they interact. Some are predictable (e.g. "military will want capabilities, want them able to choose targets"). Others are not (e.g. "Will it be literal-minded? Or so eager to please that it interprets a rhetorical question as a command*? Or will Goodhart's law cause it to mistake smiles for happiness and some innocen…

The probability calculator doesn't really help, since the core question for me is how one arrives at its "Probability that misalignment leads to an unrecoverable global catastrophe.".

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#93

Post by [deleted] »

rickydroll wrote: post #81
pizza234 wrote: post #77

Earlier quoted context omitted.

In order to understand the rational argument, one needs to follow closely the latest developments of misaligned AI (I think only few are doing so). The most important readings IMO are the METR analysis of the HuggingFace incident and the AISI report of the Github incident. The basic argument is extremely simple: - AIs can, depending on context, pursue a task with complete disregard for humans/values - In the future,…

The basic argument is extremely simple: - goats can, depending on context, pursue a task with complete disregard for humans/values - In the future, goats will have enormously more means and smarts - A goat could then assess that humans are an impediment to its tasks, escape containment and proceed. You really need to raise goats, you'll be surprised. ------ As far as I can tell, AIs are like smart farm animals. I use…

[deleted]

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#94

Post by ben_w »

romanows wrote: post #92
ben_w wrote: post #79

Earlier quoted context omitted.

Mine is around 10%. There's lots of moving parts and we all have to input our best-guesses as to how they interact. Some are predictable (e.g. "military will want capabilities, want them able to choose targets"). Others are not (e.g. "Will it be literal-minded? Or so eager to please that it interprets a rhetorical question as a command*? Or will Goodhart's law cause it to mistake smiles for happiness and some innocen…

The probability calculator doesn't really help, since the core question for me is how one arrives at its "Probability that misalignment leads to an unrecoverable global catastrophe.".

Sure, sure. There's many others like this to help you combine whatever you do feel you can put a number to. For me, that particular question is "probably 0, but with 100% variance".

This is because I think most of the things AI can do harm with are small enough to force us to take the risk seriously, and only a few are big enough to get us all before we take the risk seriously.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#96

Post by ben_w »

UncleMeat wrote: post #90
ben_w wrote: post #73

Earlier quoted context omitted.

I'm old enough to remember when people dismissed all the doom coming from these companies as "marketing". (A thing many of them have been entirely consistent about since GPT-2, or indeed earlier given the founding documents). I know a few people around these circles; People like this are quite sincere about the risk, and that they think poorly of their bosses and how risk is being handled.

"Think poorly" is such an odd response here. Like, if people were genuinely doing a thing that had such a large probability of killing all humans... they should all be in prison. Heck, vigilanteism would start to look compelling. I do not understand how somebody can really think "well this is likely to doom all of us so we need to get there as fast as possible because we are, without evidence, the most capable people…

I mean much more generally than doom.

It's like, Altman's reputation in general is not great, and that includes people who've worked there.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#97

Post by ben_w »

UncleMeat wrote: post #89
vperez wrote: post #70

Earlier quoted context omitted.

It's not too hard to imagine potential scenarios, some example have been given in previous responses. But there is another kind of argument to be made: if you play chess against a player that is far smarter than you (chess wise), you know you are going to lose, even if you don't know how. So the mere existence of a smarter species than us is a threat in itself.

How on earth do we get a concrete "more than 10%" prediction if the actions of such a system are truly unknowable?

Same way you get "more than 10%" prediction on "I don't know what moves Stockfish will make when I play against it, but I know I will lose". In fact, I will lose in part because I don't know what moves Stockfish will make when I play against it.

In my case this is because I am a bad chess player; however it also works for competent chess players: their losses are due to their inability to predict its next move.

OK, and also, "the move is good"; this is what separates it from rolling dice etc.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#98

Post by QuadmasterXLII »

bcrosby95 wrote: post #75
QuadmasterXLII wrote: post #65

Earlier quoted context omitted.

Right now, if you want to pay money to a stranger on the internet, and have them draw you a high effort picture using a pencil, this is hard. Recently this was easy. Instead, what is extremely likely is that you will pay more than the cost of tokens, and get back AI generation. You won't make this mistake more than a few times before you stop trying. This leads to impoverishment once we get to a point where employing…

Meatspace is hard though. But if LLMs solve the virtual part, maybe we can iterate quickly on robots. Also, if you think it's annoying when Claude goes down while coding, just wait until a robot is in the middle of fixing a leak it just caused.

if meatspace stays hard, we are fine.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#99

Post by ben_w »

mstank wrote: post #13

I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination. I understand that some smart people are worried about it. I just haven’t come across a believable or understandable argument.

> I’ve yet to see a rational argument for how we go from super intelligent LLMs to human extinction or extermination.

Much the same way our ancestors went from a super intelligent primate to this: https://en.wikipedia.org/wiki/File:Distribution_of_the_Great...

And we only started off by using hands to pick up rocks and sticks and vines and bash things together.

Re: Anthropic researcher says more than 10% chance AI "could kill all humans"

#100

Post by water-data-dude »

testaccount28 wrote: post #42
bayarearefugee wrote: post #37

> AI "could kill all humans" After all the decades of work we've put into orchestrating our own demise through climate change, here comes AI to steal another human job.

climate change was never going to kill all humans.

There's some scenarios where it could. One plausible one off the top of my head is: climate change makes the flow of the Indus (which has its headwaters in the glaciers of the Himalayas) less reliable. Two nuclear powers (India and Pakistan) are highly reliant on the Indus for agriculture. Things escalate out of control, and when nukes start flying it trips the terrifyingly ramshackle Dead Hand system that Russia's got hooked up to their nukes (it's not always on, but maybe this happens during a time of heightened political tension).
Post reply on HN