Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Gambling with our lives: AI researcher quits Anthropic with warning about safety

politico.eu

Post reply on HN
Print view View original

41–50 of 110 posts

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#41

Post by alpineman »

dotancohen wrote: post #18
pjc50 wrote: post #14

Earlier quoted context omitted.

This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it. Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action…

Or he believes that other labs might get there first, and he is working to counter that threat. This is the Manhattan Project again.

Or he is paid >1M USD per year

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#42

Post by -0_0- »

MrThoughtful wrote: post #25

Why would AI wipe us out? We have not wiped out apes, ants, and most other species. We even have discussions about how to actively save them from extinction.

It's not exactly trained on a neutral set of data. Engagement algorithms have ensured that a good chunk of content on the web these days is inflammatory and skewed towards the extreme (in fact in the last few months a good portion of the world has actually deemed it illegal for kids to consume because this content is so damaging to a developing brain).

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#43

Post by cplat »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

I agree with this. LLMs can actuate over capabilities we expose, so the blame isn't only on LLMs for this.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#44

Post by pjc50 »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

This feels like a distinction without a difference, like the endless wrangling over which piece of metal in a gun legally constitutes a firearm. The combination of the two may or may not be dangerous, but it's definitely the more useful combination, so of course that's how it's going to be set up.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#45

Post by Uptrenda »

Bengalilol wrote: post #7

> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them. The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move o…

They don't really care. It's just a play to tell people what they want to hear while they chase the money.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#46

Post by oefrha »

dotancohen wrote: post #18
pjc50 wrote: post #14

Earlier quoted context omitted.

This is just a bizarre thing to say that you're working on technology with that high a downside potential. If you were saying that while running a biology lab, or building a nuclear reactor, people would be demanding your head on a spike. But by not quitting it's clear that he himself doesn't really believe it. Or rather, this shows the difference between "believe" (political) and "believe" (use as a basis for action…

Or he believes that other labs might get there first, and he is working to counter that threat. This is the Manhattan Project again.

Yeah, super ethical lab earnestly believing AI has 10% chance to kill all humans within a decade and doing utmost to protect humanity is partnering with MIC and Palantir in particular. Sounds about right.

Before you tell me about how the CEO has taken a principled stand: on record, he had no problem using it against 95%+ of humanity outside the U.S., and was only against fully automatic AI killing machines, and only citing the technical reality of then-current gen tech, so one should read that as human-rubber-stamped AI killing machines are totally fine with him.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#47

Post by virgildotcodes »

dotancohen wrote: post #20
virgildotcodes wrote: post #6

At least we’ll eventually have an entity other than ourselves to blame for our annihilation.

No, the AI is still our responsibility.

I agree. It won’t stop us from othering the source of our demise as quickly as we are able.

There’s also the argument to be made that your child is their own person, and you only have so much responsibility/control over their actions.

Of course we have a good chunk of the world screaming that this child is obviously destined to be a homicidal psychopath, but what is a parent to do?

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#48

Post by azan_ »

Bengalilol wrote: post #7

> Both OpenAI and Anthropic have recently flagged incidents in which agents powered by their models went rogue I may be biased and somewhat off topic, but I see these incidents as some of the most significant of the past century. I genuinely don't understand why these companies aren't taking a smarter approach to them. The latest analyses have been, at best, laughable: identify the vulnerability, patch it, and move o…

The US government is run by idiot with dementia. No wonder govt does not take any real action.

Re: Gambling with our lives: AI researcher quits Anthropic with warning about safety

#49

Post by jonplackett »

themgt wrote: post #32

The fundamental point I think is far too often confused is the difference between LLM and agentic system. An LLM can't do anything but generate tokens. You run your LLM in vLLM or whatever, and it generates output tokens based on your input tokens. That's it! Humans then build ~deterministic systems to take those tokens and do all sorts of things with the tokens, like take actions in the real world. And then we can f…

I’m not sure what the point of this distinction is now because agents are where all the AI is now. And they’re just going to be given more and more tools and be allowed to make their own tools - or misuse things they find and turn into tools, or make them anyway without anyone noticing.
Post reply on HN