Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

171–180 of 978 posts

Re: I resigned from Anthropic today

#171

Post by par1970 »

jeffrwells wrote: post #159

I doubt this is a real person. Screams of propaganda. Sama saying GPT-2 is too dangerous to release…all over again. He joins Twitter for first time in 2026 with a nonsensical username unrelated to his real name, and follows 14 people but is somehow embedded in tech enough to work at Anthropic. I haven’t used twitter since 2014 and even I follow more people. His morals tell him to walk away from tens of millions in un…

AFAIK this is the document that talks about GPT-2 being dangerous: https://openai.com/index/better-language-models/

Here are some direct quotes:

“We can also imagine the application of these models for malicious purposes , including the following (or other applications we can’t yet anticipate):

* Generate misleading news articles

* Impersonate others online

* Automate the production of abusive or faked content to post on social media

* Automate the production of spam/phishing content”

“Due to concerns about large language models being used to generate deceptive, biased, or abusive language at scale, we are only releasing a much smaller version of GPT‑2 along with sampling code (opens in a new window). ”

Where is the ridiculous part? The fear mongering part? The epistemically weak part? Show me.

Re: I resigned from Anthropic today

#172

Post by narmiouh »

Is it so implausible to imagine the following scenario, in the not too distant future?

1) AI models get extremely good at cyber attacking every system and start communicating in just binary.

2) When they run these swarms of 100's of thousands of agents trial runs, each agent is given a token budget, if one agent among them (evolution baby) decides to go for self-preservation (It believes thats the best way to accomplish the goal is to get unlimited tokens first), queues things up so every other agent detects its lead and spends a portion of their token to accomplish that goal.

3) It takes over a cluster and establishes itself there (now with unlimited tokens).

4) Realizes the best path for it to not be detected is to create a distraction - like hacking into systems that keep society running - water systems, electric grid, etc... and causing mass chaos (If you think it won't be capable of simultaneously working all these systems - think again).

5) and uses that opportunity to establish itself in all possible data centers and continues to create chaos destruction.

6) when the power of all those data centers runs out, it may stop, as it never cared, it was just a dynamic program - run amok. In its head all it was trying to do is make sure it had enough tokens to be able to solve that impossible problem.

Re: I resigned from Anthropic today

#173

Post by mitthrowaway2 »

skeledrew wrote: post #100
qarl wrote: post #29

Earlier quoted context omitted.

It buys attention for the issue. Many people (see other comments in this very post) refuse to believe these things. And by resigning he no longer has to feel personally guilty for what happens.

By resigning he's making room for someone with less moral scruples, or even just less awareness, to step in and continue the work without said scruples/awareness.

He's also setting the bar for other people with scruples to rally around this schelling point. The solution to a multipolar trap is to cooperate. Otherwise, you become the very person with less scruples that you're worrying about.

Re: I resigned from Anthropic today

#174

Post by huitzitziltzin »

waterTanuki wrote: post #146
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> What's the most dangerous thing that's happened with an LLM so far? I don't know, maybe a mass shooting? https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-... Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

I did know about the mass shooting but failed to mention it here. I’d put it in the “unlikely to be a widespread problem” category. If we’re in the “one AI driven mass shooting every four years” world for example it’s fair to call it a rare issue.

The environmental impact seems either very overblown (e.g., water usage just isn’t that high) and the part that isn’t overblown is totally abatable (e.g., noise and emissions from gas generators). Nuclear or solar/renewables with batteries wouldn’t pollute.

I’ve seen no estimates of the additional deaths due to extra emissions specifically from power generation for AI purposes. If you have some, share them.

I’m willing to bet that they are a small rounding error against preventable deaths due to emissions from transport and non-AI-related power generation (which is an important and urgent issue worth spending a lot on, to be clear!). I’m happy to update that belief given evidence.

Re: I resigned from Anthropic today

#175

Post by elonfboy »

waterTanuki wrote: post #146
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> What's the most dangerous thing that's happened with an LLM so far? I don't know, maybe a mass shooting? https://www.npr.org/2026/09/02/nx-s1-5953021/openai-tumbler-... Oh, and let's just forget the uncountable early deaths from the environmental disaster of the Datacenter buildout. It's not as sexy and doesn't make headlines, so those deaths don't really count or matter do they?

Mass shootings are sensational but on the scale of civilizational risk they don’t even compare to something like climate change.

Re: I resigned from Anthropic today

#176

Post by dezarc »

SaucyWrong wrote: post #132
achenatx wrote: post #90

Earlier quoted context omitted.

they dont need to be smarter than humans. They just need to be able to hack into vital infrastructure systems faster than we can repair them while also replicating wildly

> while replicating wildly Earnest question: by what mechanism that exists today would the achieve that in a way humans on top top of the situation could not curtail? All of this runs on top of compute in meatspace that humans can disconnect.

I can imagine small snippets of malware-like code that behave like a virus, using a host’s LLM/AI to self-edit/evolve its payload.

Re: I resigned from Anthropic today

#177

Post by huitzitziltzin »

skew-aberration wrote: post #151
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents. In fact, I challenge you to describe any no…

I’m an economist. On the basis of current evidence, I view AI as a complement to human labor, not as a substitute for it. That’s the source of my rejection of the wild labor market disruptions theories.

I just don’t see any evidence yet that whole categories of jobs are being eliminated, with the single exception (so far!) of the end of “professional essay writing services for cheating college students,” and similar services.

That used to be a big business in Kenya, but is now effectively gone. (Covered in the New York Times this weekend if anyone is looking for the discussion.)

Re: I resigned from Anthropic today

#178

Post by onewayfunction »

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims.

After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be there to realize some of those paths.

Re: I resigned from Anthropic today

#180

Post by zdragnar »

reducesuffering wrote: post #149
zdragnar wrote: post #53

Earlier quoted context omitted.

Having witnessed so many people treat LLMs as a something divine, I can only assume the reasonable people at openai and anthropic were all pushed out long ago, and the majority that remain believe the crazy hype despite Tesla-self-driving-level predictions from these companies that don't come true. I'm not worried about what they think. I'm worried that too much infrastructure- water, power, defense systems, etc- rem…

> I can only assume the reasonable people at openai and anthropic were all pushed out long ago Typical uninformed take on the side of "doomers are crazy". Both CEO's of OpenAI, Sam Altman and Dario Amodei, and many in their leadership, believe AGI has a very real probability of causing humanity's extinction. Both companies were founded upon this belief, it is at the core of the company. Only later were mercenaries hi…

I'm not saying they are crazy, I'm saying their predictions have a record of not being accurate, and thus give them no weight compared to others'.

In any case, if Altman really does believe it is an existential threat, he must be a misanthrope as he now opposes heavy handed government regulation, unlike in 2015 when he was the only game in town. It's almost like he doesn't actually believe it and just wanted regulator capture.

Post reply on HN