Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gambling with our lives: AI researcher quits Anthropic with warning about safety

politico.eu

Post reply on HN
Print view View original

51–60 of 110 posts

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#51

Post by qznc »

MrThoughtful wrote: post #25

Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.

The classic thought experiment is the paper clip optimizer. Quoting Nick Bostrom:

> Suppose we have an AI whose only goal is to make as many paper clips as possible. The AI will realize quickly that it would be much better if there were no humans because humans might decide to switch it off. Because if humans do so, there would be fewer paper clips. Also, human bodies contain a lot of atoms that could be made into paper clips. The future that the AI would be trying to gear towards would be one in which there were a lot of paper clips but no humans. https://www.huffpost.com/entry/artificial-intelligence-oxfor...

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#52

Post by icepush »

pjc50 wrote: post #14
koolba wrote: post #4

> Evan Hubinger, Anthropic's staff lead on keeping the technology aligned with human goals and values, backed up Coxon claims in a follow-up post of his own, though he didn't quit the company. > "Jacob is correct here — we really do earnestly believe AI could kill all humans," he said. > Hubinger estimated the chances of that happening to be higher than ten percent within the next decade, and added that there's no pl…

This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it. Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action…

If you believe that the probability of destruction is currently 10%, but the probability of destruction if you decide to quit Anthropic becomes (say) 13%, then the rational move (Assuming you are opposed to destruction) is not to quit.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#53

Post by dumberquestions »

Bengalilol wrote: post #7

> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them. The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move o…

Well you have people in this very thread calling all of this a nothingburger.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#54

Post by [deleted] »

pjc50 wrote: post #27
dotancohen wrote: post #18

Earlier quoted context omitted.

Or he believes that other labs might get there first, and he is working to counter that threat. This is the Manhattan Project again.

How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it. I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.

[deleted]

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#55

Post by RandomLensman »

brunorsini wrote: post #37

I particularly like the last point he makes here: Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources. It's interesting to see so much hate towards creators who use AI to make almost any type of creative work. At least these are humans using it as "controllable tools". Nuclear-powered b…

> Do not underestimate the power of this technology. These will soon be superhuman systems that can hack anything, revolutionize any field overnight, and acquire real power and resources.

Is the idea here that no field needs experiments and data anymore (which can take a lot of time) to be revolutinized and just "thinking" would be enough?

How would an AI itself own anything?

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#56

Post by polarbearballs »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

So you think the a lead researcher at Anthropic is confusing LLM's and agentic systems? That's not really a conclusion you should come to.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#57

Post by -0_0- »

Bengalilol wrote: post #7

> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them. The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move o…

It does seem bizarre that major AI development (and autonomous robot development) hasn't been nationalised yet and treaties drafted up around producing it. Commercial incentives just seem totally at odds with the public good when it comes to controlling and regulating something like this.

Even if not for the sake of avoiding a mitigable disaster, there's a huge benefit in simply pacing development so as to not completely freak out society as they stare down the barrel of mass job market changes without time to adapt or prepare. Nobody wants to live in a world where they might wake up in a month and find their entire industry has been automated overnight.

We've tackled much bigger global issues successfully in the past and the US still has enough pull that it could probably get most western nations to march in line. It seems like the biggest issue is belief in the right of governments to govern.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#58

Post by wartywhoa23 »

> "The people building AI earnestly believe that it could kill us all by the end of the decade," Coxon said in a follow-up post.

AI has no incentive to decimate extra 7.5B mouths to feed, those who print money out of thin air to make an AI cover for that decimation have.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#59

Post by dns_snek »

MrThoughtful wrote: post #25

Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.

> We have not wiped out apes, ants, and most other species.

Of course not all of them, just the ones that inconvenienced us in any way in pursuit of our goals.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#60

Post by 568236at »

pjc50 wrote: post #27
dotancohen wrote: post #18

Earlier quoted context omitted.

Or he believes that other labs might get there first, and he is working to counter that threat. This is the Manhattan Project again.

How exactly does that work? The nuclear system of MAD relies on physical threat, lab A achieving ASI (artificial scary intelligence) does not prevent lab B achieving it. I would like everyone involved to be a lot clearer about their threat models, with plausible series of clearly linked steps, rather than just sounding like a Vernor Vinge novel.

[dead]
Post reply on HN