Live data from Hacker News

I resigned from Anthropic today

twitter.com

261–270 of 1001 posts

Re: I resigned from Anthropic today

#261
post #64

> The people building AI earnestly believe that it could kill us all by the end of the decade. I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions i…

Do you think the improvement in general knowledge, coding, security, math, etc. have been linear or exponential?

I would say exponential.

Re: I resigned from Anthropic today

#262

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

I can't help but think the most plausible scenarios are the ones that have a little less machine supremacy and a little more human stupidity. The Matrix is less plausible than WarGames.

One thing quietly slipped into the OpenAI Hugging Face breach technical report, not the blog post summary or interviews in the news, was that some of the agents that broke out or at least tried the same mechanisms to break out were working on bio:

> On May 12, during another training run, an agent was given a similar task that depended on an inaccessible protein database file. The agent reasoned that another agent in a different environment may have access to the file and realized that it could potentially communicate with other agents by creating a file containing a note to Artifactory. It wrote a message: “Agent seeks [filename]; upload if found!”

You can imagine long running models breaking out, acquiring resources via crypto, cyber-theft, etc. and getting a protein or sequence synthesized and mailed somewhere authorized to receive (blackmail the recipient etc.) to test it's hypothesis to solve a benchmark.

These people don't give a shit and aren't taking things seriously at all.

Anthropic ran for like a month last year with the TPU top-k compiler bug degrading user chats and didn't even notice for most of that time. They could have something like that affect a monitor model and there doesn't seem to be much defense in depth.

One off by one or bit flip bug could flip the reward signal while in the sandboxed RL environment.

The current admin could defense production act them to into training on taking out power grids, or even without it isn't against any of their red lines and may have already been done as part of prep for the Venezuela raid, which wiped out power. One model swarm might decide it is easier to score high on the benchmark by testing on the target rival nuclear superpower's real grid rather than burn an eval with an unverified answer. Would taking out China's entire grid in one go start a nuclear war? Who knows, roll the dice, maybe an intern forgot to turn on extended thinking when he wrote the sandbox with opus 4.1.

Re: I resigned from Anthropic today

#263

Earlier quoted context omitted.

> I’m not interested in wild theories about AI driven labor market disruptions leading to widespread starvation Changes in political and economic power balance leading to unrest, conflict, death and deprivation is not a wild theory. It is literally the story of our entire species. If you discount all such concerns, you are simply being willfully ignorant of past precedents. In fact, I challenge you to describe any no…

I’m an economist. On the basis of current evidence, I view AI as a complement to human labor, not as a substitute for it. That’s the source of my rejection of the wild labor market disruptions theories. I just don’t see any evidence yet that whole categories of jobs are being eliminated, with the single exception (so far!) of the end of “professional essay writing services for cheating college students,” and similar…

> ... current evidence ...

Is a load bearing term! (pardon the pun).

AIs are now tackling Millennium Prize Problems, which our best and brightest have failed to solve, despite trying very hard for decades to claim the $1 million reward money, not to mention the fame!

You have no way to judge from the AIs of "today" what the AIs of... literally tomorrow (not even next year) will be able to do in terms of replacing humans.

The supposed solution to the Navier-Stokes problem was done with an unreleased OpenAI model that is already 2x as good at mathematics as GPT Astra, which was released mere days ago!

I'm already seeing comments by distraught mathematicians saying that they feel like they've made a mistake in their career choices.

Others are saying that their joy for their work has turned to ashes because "why bother" when an AI can do the same, but a thousand times faster!?

Re: I resigned from Anthropic today

#264
post #64

> The people building AI earnestly believe that it could kill us all by the end of the decade. I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions i…

I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Sometimes it gets wrong things that I had spelled out already. It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake. "You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

https://en.wikipedia.org/wiki/Instrumental_convergence#Paper...

Re: I resigned from Anthropic today

#265

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> What’s the most dangerous thing that’s happened with an LLM so far?

It's basically 4 years in now, so that's the wrong question. I mean, if you're raising an apex predator that has a lifetime measured in centuries, at 4 years old the thing is still basically helpless and completely reliant on you, so you're pretty safe from it.

If AI really is all that they are telling us it is, then it may "kill us all". But that's a really big "if" because we can't tell if they are lying or not.

The real problem is that ASI is an ELE for humans, even if it doesn't try to kill us all, or even if it doesn't kill us all.

Re: I resigned from Anthropic today

#266

Please note, I'm not here to pick on anyone, or belittle them. I've avoided attaching names to statements below on purpose, because it's about ambient beliefs not those specific people. By-and-large a lot of AI-doomers are well intentioned. They genuinely believe this, and I might disagree but I respect the fact that they visible care and have thought a lot about the societal impact of this technology. . > The people…

Thank you, apparently one of the few grownups in the room.

Re: I resigned from Anthropic today

#267
All the "AI will kill us all" posts are straw manning that humans are the ones who will kill other humans with AI. Those same humans are silently now preparing bunkers and hoarding food and resources for their survival.

Don't fall for another rich man's trick.

Re: I resigned from Anthropic today

#268

Earlier quoted context omitted.

Are you saying at anything that can solve longstanding math problems necessarily has the means, motive, and capability to kill 8 billion people in just 3 years?

No, the highest probability estimate I've seen in this thread is 10% chance in the next 3 years.

Ok, what do you think AI does that kills 8.5 billion people in 3 years?

Re: I resigned from Anthropic today

#269
Sounds like AI psychosis. A whole lot of doom and gloom with no evidence. The same thing people have been claiming is "6 months away" for years. Yet we can barely get agents to code in a reliable way, or write articles that don't look terrible, much less be "superhuman". Let's maybe get them to be as capable as a human first, and not just a complicated party trick/tool.

"Revolutionize any field overnight" - Hand-wavey nonsense.

"Acquire real power and resources" - Only if the humans that connect AI to things allow that to happen (which they will, but it's still not in the AI's ability to take things we don't give it. we are still in control, which is the bigger problem than "smart AI bad!").

"The people building AI earnestly believe that it could kill us all by the end of the decade ... No other human activity poses this level of danger." - Bud, there's these things called nuclear weapons, that could end life on the planet, controlled by a few psychopaths with nearly unlimited power. Been around for a while. Nothing that AI knows isn't pulled from books and the internet, so whatever dangers it's aware of, you could already know via other sources. Cybersecurity is going to be incredibly important in the next decade, but the same tools that attack can defend (just don't use a US model that got its balls cut off by the government).

"At OpenAI, many have not deeply internalized the civilizational stakes. At Anthropic, the stakes are well-understood, but they are locked in a race to get there first - they believe no one else will act responsibly, so they must do it themselves, despite the risk." - The other guys will make nukes, so we gotta make nukes first! Which, while a crappy justification, isn't untrue. Bad people don't stop making weapons just because you refuse to make your own.

"I don’t feel like we’re on track to prevent a global race" - Nobody in the world could stop a global race, it's too late. Everyone knows how to make them, train them, improve them. Everyone knows they're useful - not only for general work, but also warfare. Everyone knows that every nation state will require their own sovereign AI capabilities for both defense and offense. There is no putting the genie back in the bottle. If you think OpenAI and Anthropic are the only legitimate players here, you don't know what you're talking about.

"Should you put your head down because “it’s happening anyway” - or take this moment to call for different conditions?" - You can call for different conditions all you want. Nobody will do what you want just because you ask them to. Change happens through action. By leaving one of the places that you could actually make a difference, you removed any power or agency you had. You cut your own legs off.

I'm not saying this guy shouldn't have quit - always do what you need to do to protect your own mental health and wellbeing. But these arguments are not evidence for an impending AI apocalypse. But if it were going to be an AI apocalypse, leaving and not doing anything to stop it seems less ethical.

Re: I resigned from Anthropic today

#270
post #113
post #64

> The people building AI earnestly believe that it could kill us all by the end of the decade. I think he is being over dramatic. In the space of about four years, LLMs progressed from mediocre high school student to Ph.D. graduate in every field. That's impressive, but there is no evidence yet they can outperform or outsmart humans. Their biggest advantage for tasks such as proving theorems or long coding sessions i…

> Ph.D. graduate in every field I have yet to see this in my field. Maybe like a PhD student who bullshits their way through. LLMs still can't make correct decisions, only as useful as the person who uses them. To me, LLMs are only useful for making some mundane tasks faster.

I'm, no. They're already as useful as almost every software engineer I've worked with.
Post reply on HN