Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

901–910 of 1001 posts

Re: I resigned from Anthropic today

#901

Post by dwaltrip »

copperx wrote: post #153
matherial wrote: post #40

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement. This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it f…

To borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.

#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it.

At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...

No one can say exactly how it will play out as the complexity increases, but the risks are becoming extremely obvious.

I personally think it's extremely unlikely to "kill everyone", but there are many outcomes far short of that which seem quite plausible and rather undesirable. Russian roulette is not a smart game.

If you read the METR report and aren't scared at all, then I'd love to know why. It would help me sleep better. So please share.

TL;DR; increasing capabilities, reward hacking, and unsafe training regimes.

Re: I resigned from Anthropic today

#902

Post by ryandrake »

gpm wrote: post #298
ls612 wrote: post #216

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

Humanity does not have a great track record for "globally coordinated, collective action to solve/prevent global catastrophe." Look at how Climate Change is going.

Re: I resigned from Anthropic today

#903

Post by red-iron-pine »

jimmytucson wrote: post #425
Kim_Bruning wrote: post #106

Earlier quoted context omitted.

I'm somewhat skeptical of some of the crazier ideas too. But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event. If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose). For now let's assume the worst that can happen is that some important/significant chunk…

> For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now. If we have to disconnect from the internet to stop some kind of mold outbreak, we can't get the weather or transfer money or access healthcare or teach an elementary school class or buy stuff from s…

that's a big problem for the modern world that depends on the internet, but boy howdy we did all of those things without the internet.

ya see they had these things called cheques...

Re: I resigned from Anthropic today

#904

Post by red-iron-pine »

DirkH wrote: post #869
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

To quote another reply that is very relevant here (Please update your worldview immediately that there is no evidence of ai and biorisk): """ One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another tr…

> These people don't give a shit and aren't taking things seriously at all.

would you say these big tech bro types are... "careless people"?

cuz that's been extensively documented, and you're right

Re: I resigned from Anthropic today

#905

Post by VCFundedGenYer »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

The statement becomes very true and very real once LLMs are applied to physical objects in the physical realm such as anthropomorphic robots.

LLMs are too weak security-wise to be contained.

Re: I resigned from Anthropic today

#906

Post by platinumrad »

deaux wrote: post #366
platinumrad wrote: post #108

Earlier quoted context omitted.

> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right con…

> write an English paragraph that doesn't make me want to claw my eyes out. LLMs are very much capable of that. Your belief in the opposite has two causes. Firstly the toupee fallacy. You don't notice LLM written text that doesn't make you claw your eyes out. Second is defaults. The huge majority of people who writes text with LLMs just uses default Claude/GPT models, and put near zero effort in making it sound human…

Sure, it might take more than a single paragraph for that grating feeling to sink in, but none of the entries from the Un-Slop Fiction Prize[1] impressed me.

[1] https://www.hyperstitionai.com/unslop

Re: I resigned from Anthropic today

#907

Post by techblueberry »

senordevnyc wrote: post #890
sicher wrote: post #604

Earlier quoted context omitted.

I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.

It's head-in-sand denial because actually facing the threat (and our total inability to stop it, imo) is terrifying, and people don't like feeling that. So they're just angry and cynical instead. Plus it makes them feel smarter for whatever reason.

What, substantively speaking, are you doing differently from someone facing the threat, or are you just on the other side of the checkbox?

There are quite a few existential risks on the table. Are you even sure you've sort ordered them correctly?

Re: I resigned from Anthropic today

#908

Post by davidguetta »

vonneumannstan wrote: post #885
davidguetta wrote: post #798

Earlier quoted context omitted.

I am not rewriting. I am saying AI is not more dangerous in my opinion that Nuclear Weapons.

You are wrong. A teenager with enough GPUs could never dream of making a nuke but they are absolutely able to run capable AIs in their moms basement. Totally different class of danger

And do what with it ?

Re: I resigned from Anthropic today

#909

Post by kbelder »

emtel wrote: post #437
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

Do you think people should continue developing AI up until the point that there is evidence that AI is facilitating biological weapons development? I think we should stop before then. But that necessarily means that there will not be evidence at the time that we stop.

We also need to stop before the AI opens a portal to hell. There's no evidence of that happening... but by the time there's any indication it's possible, it's too late.

Re: I resigned from Anthropic today

#910

Post by SepiaSapient »

dinfinity wrote: post #832
ericmay wrote: post #825

I’m not advocating for this. ~~~~~~ If you really believed these labs were going to result in the extinction of humanity and you saw it first hand, don’t you have a moral obligation not to quit your job but to go and try and kill everyone working on said project, blow it up, or otherwise stop it? I take these resignations with a grain of salt precisely for that reason. If I worked at a job and I saw some crazy guy or…

> I’m not advocating for this. You're not advocating for it because murdering people is highly immoral and illegal. That answers your question for the most part. You also have to remember that the magnitude and speed of this stuff is very new to pretty much everybody . It's not exactly trivial to go from "I'm just doing my job" to "I need to kill everybody in my company to save the world" in a year or two, especially…

This is a non sequitur. Yes, if someone doesn't actually believe in imminent AI doomsday they are not going full Sarah Connor on Anthropic. Yes, HN will not like if someone pipe bombs the office. What does it have to do with @ericmay's point?

He directly says "They are racing straight to self-improving superintelligence and gambling with our lives", that's a claim of imminent threat to humanity. If he truly believes that, direct action shouldn't be off the table nor even be immoral. Obviously he might not have the temperament or simply be a pacifist, that's totally normal but the point is that tweets are weird even then.

Why just make some tepid tweets if you think humanity is at risk? Why doesn't he leak documents and messages with the unfiltered opinions of leadership? Why isn't screaming "we are going to die" in CNBC? You'know, try anything?

That's where the "take it with a grain of salt" is important, maybe this is just a jumping point to a better job in the next super safe AI lab (for realsies this time)

Post reply on HN