Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

921–930 of 983 posts

Re: I resigned from Anthropic today

#921

Post by red-iron-pine »

rurban wrote: post #620
skybrian wrote: post #241

Earlier quoted context omitted.

The mathematics research results are certainly impressive, but I don't see what that has to do with robotics. Waymo is getting somewhere, but it's been a long slog. There doesn't seem to be much progress on, say, package delivery. For more, see: https://secondthoughts.ai/p/14-reasons-robotics-is-hard

Didn't you follow the robotics advances in the Ukrainian - Russian battlefield? There is now a death zone of 50km, only controlled by drones and automatic weapons.

50km is just the "very likely to get hit" part of the gray zone

ISR drones and other tools can go as far as 200km, albeit with worsening visibility.

in most cases everything meaningful in that 200k is seen, and the only limitation is range of the drones, artillery, missiles, and potential countermeasures.

Re: I resigned from Anthropic today

#922

Post by dwaltrip »

copperx wrote: post #153
matherial wrote: post #40

Unlike most other commenters, I applaud him for acting on his principles. If you sincerely believe that, of course you should act. You might not succeed, but your voice might be the one that tips the scales and starts a broader movement. This doesn't mean I agree with him. The fears of doomsday caused by rapid takeoff have been with us since day 1 and the mechanism is always basically "AI invents magic that sets it f…

To borrow on the 1990s Slashdot meme: 1. Invent transformer architecture. 2. Scale it up. 3. ??? 4. Machines become sentient and kill us all. OpenAI and Anthropic pinky promise that they have figured out #3 and they're not BSing just to get more funding, no. But because we live in a culture of fear, everyone eats it up no questions asked.

#3 is actually decently well mapped out. You just don't find it plausible or misunderstand it. I'd appreciate if you wrote your actual arguments against it.

At lower capability levels, the patterns are very clear and have been studied to death. E.g. Why LLMs say they have correctly fixed a broken test when they haven't. What we saw with Hugging Face is literally the exact same problem, just scaled up and with more capable agents. This shit was predicted decades ago...

No one can say exactly how it will play out as the complexity increases, but the risks are becoming extremely obvious.

I personally think it's extremely unlikely to "kill everyone", but there are many outcomes far short of that which seem quite plausible and rather undesirable. Russian roulette is not a smart game.

If you read the METR report and aren't scared at all, then I'd love to know why. It would help me sleep better. So please share.

TL;DR; increasing capabilities, reward hacking, and unsafe training regimes.

Re: I resigned from Anthropic today

#923

Post by ryandrake »

gpm wrote: post #299
ls612 wrote: post #216

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

Humanity does not have a great track record for "globally coordinated, collective action to solve/prevent global catastrophe." Look at how Climate Change is going.

Re: I resigned from Anthropic today

#924

Post by red-iron-pine »

jimmytucson wrote: post #428
Kim_Bruning wrote: post #106

Earlier quoted context omitted.

I'm somewhat skeptical of some of the crazier ideas too. But the hugging face incident was actually very large. It was not a single agent, it was not a single target, and it was not a single event. If nothing else, that's a bit of a warning as to what can happen next time (By accident, or if a government decides to go on purpose). For now let's assume the worst that can happen is that some important/significant chunk…

> For now let's assume the worst that can happen is that some important/significant chunk of (transitively) internet connected stuff goes haywire all at once. That's probably your upper limit of what can go wrong for now. If we have to disconnect from the internet to stop some kind of mold outbreak, we can't get the weather or transfer money or access healthcare or teach an elementary school class or buy stuff from s…

that's a big problem for the modern world that depends on the internet, but boy howdy we did all of those things without the internet.

ya see they had these things called cheques...

Re: I resigned from Anthropic today

#925

Post by dinfinity »

ericmay wrote: post #894
dinfinity wrote: post #877

Earlier quoted context omitted.

> if someone was 100% certain This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans. You're arguing from an irrelevant hypothetical (in which, I might add, most people would still not automatically change into murderous psychopaths as you apparently thi…

> This is unreasonable because nobody can know 100% certain that the AI built at Anthropic will kill everyone. It is also a big-ass strawman because the author never claimed he was 100% certain that AI will kill all humans. Ok then stop arguing about my hypothetical if you don't want to engage with it. Those of us who are interested in talking about it can do so in peace without distracting comments. > change into mu…

I'm not the one reacting to reasonable behavior with some edgy comment containing broken logic.

The author of the Twitter thread is not some egotistical attention seeker and never said "omg I’m quitting they’re going to kill everyone". His logic for quitting his job and not acting like a psychopath is sound. Yours is not.

Re: I resigned from Anthropic today

#926

Post by red-iron-pine »

DirkH wrote: post #889
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

To quote another reply that is very relevant here (Please update your worldview immediately that there is no evidence of ai and biorisk): """ One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio: > On May 12, during another tr…

> These people don't give a shit and aren't taking things seriously at all.

would you say these big tech bro types are... "careless people"?

cuz that's been extensively documented, and you're right

Re: I resigned from Anthropic today

#927

Post by hegelstoleit »

skeledrew wrote: post #904
hegelstoleit wrote: post #756

Earlier quoted context omitted.

You missed the point. > Just because someone else might be willing to do it isn't a reason to continue doing it. Hence, someone filling the position after you leave isn't a reason to continue doing it, whereas the fact that it's an immoral thing to do is a reason to stop doing it. Someone filling the position after you leave has no bearing on the reasons that are relevant to what you ought to do here. It's like decid…

> the fact that it's an immoral thing to do This is carrying a lot of weight. What makes working on frontier models "immoral"? Because there is potential for misuse? Would you also say working at a knife factory is immoral? Or even at a nuclear or biological pathogens research lab? After all those too can be misused. And if one does suspect a workplace is moving in a bad direction, and there's no immediate danger to…

Well we're not debating the morality of it, the question is what he should do, given that he believes that it's immoral.

In your analogy the correct thing to do would be to call the authorities to help. The problem with the analogy is that there isn't a 911 number to call for this scenario. You need to appeal to the people who have the power to do something about the situation. Quitting in a very public way is one way to get someone with that authority to do something about it. I don't really see a better alternative.

Re: I resigned from Anthropic today

#928

Post by VCFundedGenYer »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

The statement becomes very true and very real once LLMs are applied to physical objects in the physical realm such as anthropomorphic robots.

LLMs are too weak security-wise to be contained.

Re: I resigned from Anthropic today

#929

Post by platinumrad »

deaux wrote: post #368
platinumrad wrote: post #108

Earlier quoted context omitted.

> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right con…

> write an English paragraph that doesn't make me want to claw my eyes out. LLMs are very much capable of that. Your belief in the opposite has two causes. Firstly the toupee fallacy. You don't notice LLM written text that doesn't make you claw your eyes out. Second is defaults. The huge majority of people who writes text with LLMs just uses default Claude/GPT models, and put near zero effort in making it sound human…

Sure, it might take more than a single paragraph for that grating feeling to sink in, but none of the entries from the Un-Slop Fiction Prize[1] impressed me.

[1] https://www.hyperstitionai.com/unslop

Re: I resigned from Anthropic today

#930

Post by techblueberry »

senordevnyc wrote: post #911
sicher wrote: post #613

Earlier quoted context omitted.

I'm also baffled. AI that is substantially smarter than us is a very potential threat to us - and we won't even be able to comprehend what most of those threats may be.

It's head-in-sand denial because actually facing the threat (and our total inability to stop it, imo) is terrifying, and people don't like feeling that. So they're just angry and cynical instead. Plus it makes them feel smarter for whatever reason.

What, substantively speaking, are you doing differently from someone facing the threat, or are you just on the other side of the checkbox?

There are quite a few existential risks on the table. Are you even sure you've sort ordered them correctly?

Post reply on HN