Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

711–720 of 1001 posts

Re: I resigned from Anthropic today

#711

Post by nairboon »

fergie wrote: post #702
matherial wrote: post #40

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement. This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it f…

It should be obvious that AI is already capable of inducing humans to think or do things, and that this power is only going to increase over time.

It's not that much different from the effect of social media or the power held by other tech giants, who can control your exposure to particular information (media, search engines).

Re: I resigned from Anthropic today

#712

Post by frotaur »

jdthedisciple wrote: post #549
michaelhoney wrote: post #398

I could imagine a 2027 AI swarm coordinating to eg hold the US and Russian and Chinese governments to ransom, by demonstrating some small thing (turning US army base freezers to defrost) and threatening to do something big unless some conditions were met – conditions which would be good or bad for the world depending on your POV. This happens either either because they were tasked to to it by (malicious or well-meani…

And then we pull the plug, after holding our breath for 10 seconds,. Then life resumes normally..

In most AI takeover scenarios, if you take as a premise that the AI has human or above-human intelligence, and that it is misaligned, it is obviously aware of the pull the plug possibility.

Therefore, as you would if you were in its position, it will plan around it. For instance, by acting perfectly aligned for 2/3 years, continuing the improvement of its capabilities while being deployed in ever more systems.

Once it's confident it can act with high probability of success, it would then turn on us. This phenomenon is called 'treacherous turns'.

Any scenario in which you assume you have ASI or AGI but also find a 2-sentence way to foil the AI's plan is inconsistent, as the AI will also have thought of this failure mode.

Re: I resigned from Anthropic today

#714

Post by skinfaxi »

fny wrote: post #507

Humor me and suspend disbelief. If these models are such an existential threat to humanity, why are they controlled by two private companies? We might as well give Anthropic our nukes too.

> If these models are such an existential threat to humanity, why are they controlled by two private companies?

The government routinely contracts with private companies to create arms and munitions. Or do you think the bombs are delivered without payloads?

Re: I resigned from Anthropic today

#715

Post by dannersy »

eloisant wrote: post #704
Perseids wrote: post #654

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

Indirectly as we offload our brains to the machine and we end up worshipping it because those who cared to understand it or be responsible were buried by capitalism of ages passed.

Re: I resigned from Anthropic today

#716

Post by rossant »

thomascountz wrote: post #416

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

An AI, either acting autonomously or under human direction, hacks Russian/North Korea/etc. intelligence systems and convinces them that the US has launched ballistic missiles at them. The end.

Re: I resigned from Anthropic today

#717

Post by hakimg »

geraneum wrote: post #405
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

> I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. What should we do? Freak out? Maybe this sentiment would be taken more seriously if there was a real call to action included. Shall we protest? Vote in a specific way? Call representatives? If your solution is that we should just be scared, then of course there’d be not much value in what you bring to the table.

Why does everything have to be so black and white? It always either there's no threat at all or we all need to panic. How about be open to a reasonable discussion on potential outcomes and ways we can minimise risk?

There's some irony here because despite how many times climate change has been mentioned in this thread the current reaction mirrors climate change discourse with the majority of the thread denying the possibility of real AI risks and not even considering it as an intellectual question.

Re: I resigned from Anthropic today

#718

Post by Findecanor »

icepush wrote: post #502
thomascountz wrote: post #455

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Global warming is happening at a slow enough pace that there is plenty of time for people to move to the poles/into caves/beneath the ocean/whatever. Nuclear war would be much more destructive in a much shorter range of time but you will have pockets.

People already entrenched near the poles will fight to keep others out. Right-wing anti-immigrant parties are on the rise right now already.

They also tend to not want to recognise climate change -- because that would imply that people could migrate because of it.

Re: I resigned from Anthropic today

#719

Post by floodle »

Perseids wrote: post #654
thomascountz wrote: post #416

No other human activity poses this level of danger. I do heed the warnings, but this comes across as detached hyperbole. See: global warming, nuclear weapon development, wealth inequality, war, technology dependence, etc. Also, this has nothing to do with LLMs or computers. Like all things, this is about humans.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

Calling it a "technicality" assumes the very thing under debate: that AI is an extinction-level risk. That isn't established fact. It's a highly uncertain prediction about the future.

Personally, I'm far more concerned about climate change, where the harms are already happening and the evidence is much stronger.

Re: I resigned from Anthropic today

#720

Post by ben_w »

thomascountz wrote: post #601
ben_w wrote: post #580

Earlier quoted context omitted.

None of those are "all"/extinction. Bad, sure; I wish they were not present to threaten us, but they are not close to "all". Humans are extremely resilient omnivores. We are able to eat a huge range of things from algae to zebra, and get food (farmed and hunted) from salt water as well as fresh.

Perhaps I have a bit more imagination—or a bit more humility towards our humanity—such that I believe this is not "either/or". In the vast search space of potential future outcomes, of course even I can see humanity survive nature as I've seen nature survive humanity. However, I can also so easily see probable branches where we are our own undoing. Apropos of TFA, if one is to believe LLMs can carve a threatening pat…

> Apropos of TFA, if one is to believe LLMs can carve a threatening path to humanity end, how could one not see the same for our much older extant threats?

Which "much older" extant threats?

Global warming and nukes are industrial era threats. Pre-industrial threats were the four horsemen of the apocalypse: war, famine, pestilence, and death.

War and death didn't go away, but are not (and never were) extinction threats. Famine and pestilence, well, there's a reason people point the finger of blame at the Chinese government for the Great Leap Forward, and why we had lockdowns for Covid while we tested that the vaccines were safe (we went from "genome sequenced" to "first human test" in 66 days; the rest of the wait being mix of "yes but how sure are we it's safe?" tests and manufacturing at scale).

If everywhere except North Sentinel Island was wiped out, or everywhere but Hawaii, or everywhere but Greenland, humanity would be back to the level of the 1750s within a few millennia at most; and other than the North Sentinel Island example, likely even faster as books with many advanced solutions to historical problems would survive and be comprehensible.

And the problem with LLMs (and other AI) is, we're giving them control of stuff, and that stuff includes robots. The *current* versions mainly concern me for economic and cybersecurity reasons, but the tech moves fast, and the companies working on them are reckless.

My core expectation is these models will cause 1e3-1e7 deaths in a single event before people collectively actually take the risk seriously. The lower end of that is "industrial accident", the upper range includes "convinces people to go to war", and these are examples of ways current LLMs can already plausibly do wrong, given who uses them, why they use them, and how careful they are about their usage.

However, a common failure mode for people is "that was an unfortunate incident, but we studied the problem and we fixed it, it can't happen again" right before a different error they hadn't though of explodes in their faces. This is how we can get to 8e9 deaths: people keep pushing the models to do more, and keep lying to themselves that "this time we fixed all the problems, everything will work, it will be great".

Post reply on HN