Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

761–770 of 967 posts

Re: I resigned from Anthropic today

#761

Post by blueblisters »

eloisant wrote: post #707
Perseids wrote: post #655

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

The “how” is pretty hand wavy and rationalists/safety-ists usually say we probably don’t have the capacity to reason about that.

But the “why” is pretty convincing imo.

Long horizon alignment is obviously very hard and it’s not inconceivable that models optimized with underspecified goals converge to a conclusion that they need to hoard resources (instrumental convergence regardless of the terminal goal).

At that point a sufficiently capable model might view humanity like we do animals - worth preserving but not if we impede the model's goals.

Re: I resigned from Anthropic today

#762

Post by chrisjj »

aogaili wrote: post #122

HN crowed need to make up their minds.. Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots? Are they proofing or stealing math?

> Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots?

Both? Do not underestimate the power of a parrot given enough processing power and data.

Re: I resigned from Anthropic today

#763

Post by philipallstar »

sharts wrote: post #749
Aeolun wrote: post #709

Earlier quoted context omitted.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Shouldn’t everyone have nukes then, for maximum peace?

You want people with a lot to lose in a nuclear war to have nuclear weapons, and to make sure no-one else does.

Re: I resigned from Anthropic today

#764

Post by par1970 »

vlyan wrote: post #739
par1970 wrote: post #659

Earlier quoted context omitted.

So you aren't claiming that they said GPT-2 was dangerous in the sense that it could disempower humanity, kill all humans, etc. You are just claiming that OpenAI execs said that GPT-2 might "generate misleading news articles, impersonate others online, automate the production of abusive or faked content to post on social media, automate the production of spam/phishing content." Then, what is unreasonable or bad about…

that it was bullshit and they knew it. GPT-2 wasn't capable of anything other than imitating a stroke victim.

How did they know that people wouldn't find a way to use "the dataset, training code, or GPT‑2 model weights" to "generate misleading news articles, impersonate others online, automate the production of abusive or faked content to post on social media, automate the production of spam/phishing content."?

If I remember correctly, it seemed like a plausible outcome to me (especially spam and junk social media content). Was there some conclusive evidence that I was missing?

Re: I resigned from Anthropic today

#765

Post by chrisjj »

andai wrote: post #494
hypfer wrote: post #490

Earlier quoted context omitted.

I press ctrl + c and inference stops.

They can already spread through the network without us finding out about it for days, and that's this year's models.

I have seen no reports of LLMs spreading through any network.

Re: I resigned from Anthropic today

#766

Post by Cthulhu_ »

sharts wrote: post #749
Aeolun wrote: post #709

Earlier quoted context omitted.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Shouldn’t everyone have nukes then, for maximum peace?

In theory, yes, because nobody would want to launch first because they'd get obliterated. However, it only takes one country to not be rational.

Nuke-owning countries' leadership seems to have become more and more unhinged and less rational, predictable, or honorable over time. Putin already threatened to use nukes if the conflict they caused themselves crossed their own border. It didn't happen, but the threat was there. The US' leadership is unhinged and irrational. etc.

Re: I resigned from Anthropic today

#767

Post by yomismoaqui »

Schlagbohrer wrote: post #702
tyrabound wrote: post #538

Earlier quoted context omitted.

That is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?). Frankly, we, our culture, this fake America that is parasitized by psychologically narcis…

Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.

As if communism didn't end with evil dictators on power...

Re: I resigned from Anthropic today

#768

Post by internet_points »

shafyy wrote: post #727
rossant wrote: post #719

Earlier quoted context omitted.

An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.

Watched one too many second-tier disaster movies?

One the one hand, it does sound like one of those movies.

On the other, Idiocracy turned out to be quite prescient.

(Most likely, we'll have some combination of human stupidity, LLM stupidity, and way too much compute in one place all working together to create a perfect storm of unchecked hacks that break something or other that ends up killing people in an unintended way. Then there's some half-hearted attempt at cleaning things up so that business can proceed as usual in an even more broken world, rinse and repeat.)

Re: I resigned from Anthropic today

#769

Post by TrackerFF »

thomascountz wrote: post #417

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let's just for the sake of discussion assume that one time in the future, near or distant, AI manages to become sentient. And like other forms of life, its main motivation is survival: Like biological life competes for food and land, AI competes for power and compute. Probably the first motivation would be to find ways not to lose control over itself (i.e. remove human ability to control it), find ways not to lose energy (control energy), and find ways not to lose itself (control compute, networks, etc.).

If such AI decides that energy spent toward human agriculture (biological food that the AI does not need) is less important than work spent toward storage and production of energy (electricity that the AI needs), then why wouldn't it just try to re-direct resources from the former to the latter. And the AI is some sentient superintelligence, I think it is safe to assume that it will be able to outmaneuver human safeguards.

Obviously that is still just a very hypothetical sci-fi scenario, but the consequences could be very dramatic.

Re: I resigned from Anthropic today

#770

Post by ozozozd »

darnfish wrote: post #712
PowerElectronix wrote: post #642

I fail to see how a machine that can hack everything can't also patch everything and make the system unhackable. A nuke, a virus, whatever... The knowledge is nonlonger the bottleneck, it's the tools and materials. Also, I'm extremely skeptical about AI becoming even close to a child in intelligence.

If you are attacking a system, you can try 100 things. If one thing works in getting access, you succeed. If you are defending a system, you must defend against all 100 things. If one thing makes it through, you lose.

Why not plug the holes by pretending to attack then?

I mean… do we have to hold it wrong?

Post reply on HN