Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

861–870 of 1001 posts

Re: I resigned from Anthropic today

#861

Post by OccamsMirror »

shepherdjerred wrote: post #227
slicktux wrote: post #201

Earlier quoted context omitted.

Why are we putting so much weight (no pun intended) on AI companies. At the end of the day the scaled up LLM transformers lack emotion and will… They do as they are told; or more correctly put. They do as they are programmed to do so.

Why are you bringing emotion and will into this? Does something have to have those to be useful or dangerous? > They do as they are told; or more correctly put. They do as they are programmed to do so. _Nobody_ told them to hack Hugging Face. Do you really not understand what is happening?

The training (i.e. all of the red teaming and CTF content they could scrape) told them to.

Re: I resigned from Anthropic today

#862

Post by OccamsMirror »

jenadine wrote: post #318
cure_42 wrote: post #255

Earlier quoted context omitted.

That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's…

I'm order to guess the next token in a love poem, they must understand love. In order to predict the next token in a chess game between grand master, they must master chess. In order to predict the next token in a computer program, they need to be able to program anything. They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.

LLMs are famously bad at Chess. They have no world model and they are not trained to be good at Chess. This is not the amazing point you think that it is.

Re: I resigned from Anthropic today

#863

Post by treszkai »

Aeolun wrote: post #686
Mizza wrote: post #683

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Except in the case of nuclear weapons, humans have their hands on the launch codes, and the nukes are not going to launch themselves. In the case of extremely capable AI, nobody has their hands anywhere close, because nobody is able to – nor willing to – supervise a thousand agents running in parallel, when they usually get shit done. Except eventually they are so good at getting things done that they disregard the values of the human asking them the request, such as "do not hack other corporations". Interpreting human intent precisely is an extremely hard problem, especially in the face of something 1000x faster than you operating autonomously.

Re: I resigned from Anthropic today

#864

Post by ericmay »

strogonoff wrote: post #852
ericmay wrote: post #815

I’m not advocating for this. ~~~~~~ If you really believed these labs were going to result in the extinction of humanity and you saw it first hand, don’t you have a moral obligation not to quit your job but to go and try and kill everyone working on said project, blow it up, or otherwise stop it? I take these resignations with a grain of salt precisely for that reason. If I worked at a job and I saw some crazy guy or…

Consequentialism is not the only option. It invaded the mainstream thanks to effective altruism and friends, but deontological ethics is alive and well. Quitting your job instead of blowing up the office is well aligned with the latter.

> Quitting your job instead of blowing up the office is well aligned with the latter.

How so, in your view?

Re: I resigned from Anthropic today

#865

Post by program_whiz »

program_whiz wrote: post #843
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

What? Shutting down infrastructure, destroying critical records (economic collapse), spoofing / impersonating world leaders (confusion, panic, world war, nuclear event), taking over comms generally (you get a message to evacuate your home due to impending disaster, is it real?). All these could cause sufficient chaos and fear as to instigate global collapse. Once the panic sets in after a few of these indicidents, un…

I wrote about this, as I thought it warranted a deeper look: https://nonlineartransform.substack.com/p/world-ending-ai-to...

Re: I resigned from Anthropic today

#866

Post by ericmay »

zeven7 wrote: post #848
ericmay wrote: post #824

Earlier quoted context omitted.

> You're not advocating for it because it is highly immoral and illegal. That answers your question for the most part. Well truthfully I’m not advocating for it because I don’t have any insight into these labs or the true capability of these models. But as a hypothetical if someone knew with 100% certainty that Big AI Button was being built and it would kill all humans or destroy all of humanity there is no moral amb…

What if you have 10-20% certainty the AI being built will kill everyone? Go on a rampage and land yourself in prison just so Lab B or China can win the arms race and their AI can kill everyone? Probably not. Quit your job? Sure.

What if you have a 70% certainty?

You don't have to go on a rampage in an American lab. If Chinese labs were ahead I'd bring about the same discussion.

Though separately if the United States (or China or anyone) believed one country or another was truly going to achieve something akin to a metaphorical AI Supremacy maybe you nuke them, or at least the labs/researchers. Many a sci-fi movie has been built on a similar "first strike" premise.

Re: I resigned from Anthropic today

#867

Post by sillyfluke »

frabcus wrote: post #660
gizmodo59 wrote: post #495

Earlier quoted context omitted.

This is a lame argument. Questioning his financial incentive is very legit. Why should we trust him?

Questioning his finances is a poor argument because he clearly would have more money if he had stayed at Anthropic for the next couple of years, than he will have by leaving. Even if he still has vested options, or savings from his salary or whatever, those would be larger if he stayed.

> clearly

Debatable, and clearly not clear. It's weird to say this when there are obvious counter examples staring us in the face, not least of all the founder of the company he just left, of who (Dario) people could have said the same thing when he left OpenAI.

Granted, there is less likelihood now then there was then but this much is clear: if he starts his own company, he may make more.

Re: I resigned from Anthropic today

#868

Post by ls612 »

gpm wrote: post #296
ls612 wrote: post #215

Earlier quoted context omitted.

Historically it almost always takes actual fatalities rather than near misses to ground an aircraft, and aviation is famous for its obsession with safety compared to other industries.

Airplanes have pretty bounded damage. Generally you kill at most a few hundred people. Even weaponized a few thousand. This is a risk profile that allows risk taking with near misses and waiting until something goes wrong to fix it (though doing so is rightfully uncomfortable and frequently unethical). The people worrying about AI risk are worrying about "it goes wrong once and kills billions of people". That's not a…

The story of the safetyists has the convenient property of being unfalsifiable, so they can always claim doom is just around the corner. It’s the secular/EA version of the second coming.

Re: I resigned from Anthropic today

#869

Post by skrellm »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> I really, really disagree with that statement.

But you're wrong, although not for the reasons you think (suggested by your examples).

> I don’t think ai models come close to nuclear weapons

Unless AI models are controlling the drones/icbms carrying those. https://www.theguardian.com/technology/2026/aug/18/autonomou...

> or to run-of-the-mill, everyday carbon emissions

Unless AI models are responsible for those said emissions. https://arstechnica.com/tech-policy/2026/08/amazon-funds-big...

> What’s the most dangerous thing that’s happened with an LLM so far?

That it's built solely on "trust me, bro"?

> right now I am not concerned at all.

You should be. There's no scenario where this ends well: if AI labs succeed, we all will loose our jobs. If they fail, then the bubble bursts. Economic crisis is inevitable either way, and that's just the tip of the iceberg.

Re: I resigned from Anthropic today

#870

Post by eudamoniac »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

There's no evidence building the torment nexus will be bad, actually! It's just speculation!
Post reply on HN