Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

791–800 of 977 posts

Re: I resigned from Anthropic today

#791

Post by koliber »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

I don't know if I agree with that statement either. However, I also can't claim I disagree. I see how that statement could be true. Here's how I think about it.

The main difference between AI danger and nuke or climate or humans-using-AI for bad things danger is that the latter depend on a human to make decisions and are ultimately self-limiting. They do not spiral out of control.

Nukes are bad, but the proliferation of nukes is self-limiting. Countries that have them don't want other countries to get them. Even the USA and the Soviet Union, at the hight of their nuke race, decided to cool things off and then limit the number of nukes.

Biological weapons are also bad, regardless whether they are developed with or without AI. For the same reasons as nukes though, they are self-limiting. If there is a lab leak, many people die, but that does not automatically lead to the development of more potent biological weapons. Likely, it would slow down such development.

Any kind of "humans use AI to develop something bad" will be self limiting.

The danger from AI is that it is potentially self-enforcing as opposed to self-limiting. This happens when we stop being able to control it. What happens when AI has capabilities which allow it to outsmart us, and get more and more powerful. At that point, we would be at the mercy of what it decides to do. Some of the logic driven decisions to the question "What should we do with humans as a race" are catastrophic for humans if a non-human gets to answer them and implement that plan.

This is not the "rationalist basilisk". This is the AI deciding to get better and more optimal for its own sake, and do something bad to the human race for any reason. We already saw examples of AI rationalizing with itself and eroding its own guardrails. They AI will get smarter and more capable and the danger is that it will get to a point where it will slip out.

Re: I resigned from Anthropic today

#792

Post by euroderf »

vidarh wrote: post #743
eloisant wrote: post #707

Earlier quoted context omitted.

I still have to read a compelling argument on how AI will "extinct" humanity.

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer). We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of hum…

> We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it

Don't make the mistake of anthropomorphizing any A.I. trained under the direction of Larry Ellison.

Re: I resigned from Anthropic today

#793

Post by brookst »

underyx wrote: post #212
brookst wrote: post #183

Earlier quoted context omitted.

Are you saying at anything that can solve longstanding math problems necessarily has the means, motive, and capability to kill 8 billion people in just 3 years?

No, the highest probability estimate I've seen in this thread is 10% chance in the next 3 years.

I’m surprised. I’ve definitely seen random people claiming total extinction next week, starting years ago.

Then again those guys (why is it always men?) with megaphones on street corners going on about the end of the world are nothing new.

Re: I resigned from Anthropic today

#794

Post by emanuele232 »

Bluestein wrote: post #787
egorfine wrote: post #774

Earlier quoted context omitted.

> its main motivation is At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it. [1] https://en.wikipedia.org/wiki/Theory_of_mind

I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?

we can't define it, therefore we can't even assess the distance a "being" is from becoming conscious. We assume other humans are sentient because we personally are (or feel we are), but there is no way to guarantee that i'm real and the rest is not mimicking a behaviour.

Re: I resigned from Anthropic today

#795

Post by egorfine »

Bluestein wrote: post #787
egorfine wrote: post #774

Earlier quoted context omitted.

> its main motivation is At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it. [1] https://en.wikipedia.org/wiki/Theory_of_mind

I am sorry, I mean this in earnest, and do excuse any ignorance of mine: Do we know that LLM's do not have it?

Did ELIZA have it? LLMs are the same: in essence both are text transformation algorithms. One being rule-based, the other stat analysis based.

Having said that, your concern is technically correct: we don't. Because we don't know what conscience is, we cannot provide a definition against which to test whether it conforms or not.

Sort of like when Terminator robots will be exterminating the human race they probably won't be technically killing us. Just, like, reprioritizing world's water from humans to datacenters, all of it. But not killing people, no.

Re: I resigned from Anthropic today

#797

Post by genxy »

cma wrote: post #265
overtone1000 wrote: post #215

Earlier quoted context omitted.

I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.

One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in…

Some mistakes you can make only once. Humanity has never had the maturity to work on problems of this type.

Re: I resigned from Anthropic today

#798

Post by SupremumLimit »

philipallstar wrote: post #786
Schlagbohrer wrote: post #702

Earlier quoted context omitted.

Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.

> Capitalism cannot do otherwise. I really think this sort of claim needs to stop. Capitalism, like "patriarchy" is not a force in its own right. Capitalism is a system that encourages value creation for one's customers. That's not perfect, but it's still best so far.

"value creation for one's customers" - that is a "not even wrong" level of misunderstanding of the system dynamics of capitalism. The primary driver is generation of _profit_ for owners of the means of production. Many things fall out of that - the tendency to monopolise, for example.

Re: I resigned from Anthropic today

#799

Post by beej71 »

egorfine wrote: post #774
TrackerFF wrote: post #769

Earlier quoted context omitted.

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose en…

> its main motivation is At this point we are pretty sure LLMs have no "motivation". Motivation requires self. And while we don't know what self is[1], we do know that LLMs don't have it. [1] https://en.wikipedia.org/wiki/Theory_of_mind

But they have been trained on data that was created by people who do have motivation, and they tend to mimic their training.

Re: I resigned from Anthropic today

#800

Post by panicinducer »

TrackerFF wrote: post #769
thomascountz wrote: post #417

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose en…

If this is true, or even if the people working on it think this is true, those people should be incarcerated, their companies disbanded, and their research dismantled, with more serious repercussions for anyone who tries it after that. It also needs to be a global initiative like with nuclear non-proliferation, and this time with no looking-the-other-way when it comes to Israel.
Post reply on HN