Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

741–750 of 1001 posts

Re: I resigned from Anthropic today

#741

Post by vidarh »

eloisant wrote: post #705
Perseids wrote: post #654

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

Re: I resigned from Anthropic today

#742

Post by ak39 »

areoform wrote: post #243
threatofrain wrote: post #213

Earlier quoted context omitted.

Before we discussed how important security was, we got insurance, we made libraries and products, we used compliance software, etc. Except how honest were we about all that stuff? How much risk was actually in the air, and what was keeping us accountable on security in either direction of over or under-investment? Now a reckoning is here. The potential to be attacked might actually translate to being attacked.

People have died due to ransomware attacks on hospitals. Powerplants have been attacked. Stuxnet and industrial control malware exists. What will the AI do that hasn't been tried before?

Quite correct with that question.

I believe the universal answer is: incompetent malicious actors are now capable too. Which implies the pool from which to draw the intersection between capable and malicious has grown. (That's my reading)

Re: I resigned from Anthropic today

#743

Post by entropyneur »

aogaili wrote: post #12

He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race. His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.

What do your expect him to do? Blow up the office Miles Dyson style?

Re: I resigned from Anthropic today

#744

Post by BartjeD »

computerphage wrote: post #471
thomascountz wrote: post #455

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

Global warming causing war between nuclear armed opponents. Migration flows and competition over ever shrinking resources.

It's silly to treat these things as separate categories as if only one will happen at a time. These issues are happening, and are going to happen all at once. That is the fundamental challenge. AI plays into that because it can make wars and nuclear exchanges so much more effective.

If an AI tells the President the USA can survive China's nuclear barrage mostly unschathed, perhaps he'll press the button...

Re: I resigned from Anthropic today

#745

Post by surgical_fire »

Eueudhsbsj32 wrote: post #285
surgical_fire wrote: post #142

Earlier quoted context omitted.

I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Sometimes it gets wrong things that I had spelled out already. It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake. "You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

> I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Can you give a simple example? I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.

Yes. Yesterday, Opus 5 on Claude Code with high effort.

It was to build an extremely simple job using an internal framework to walk through a table and log the ids os some records that have a certain scenario.

There's a ton of jobs exactly like this in the codebase, and the framework code is in the codebase as well.

It was so silly I even thought of writing it myself, probably took me longer to steer claude to do it for me.

Anyway, it refused to use a method from the framework to retrieve the parameter as a list, it wanted to retrieve it as a string and parse the commas. I had spelled out in the initial prompt what method it should use.

I really don't like Claude much. Frontier my ass.

Re: I resigned from Anthropic today

#746

Post by xelxebar »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

Your call for clarity has merit, but I think you have backed yourself into a corner, honestly.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

Re: I resigned from Anthropic today

#747

Post by sharts »

Aeolun wrote: post #707
Mizza wrote: post #704

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Shouldn’t everyone have nukes then, for maximum peace?

Re: I resigned from Anthropic today

#748

Post by input_sh »

huitzitziltzin wrote: post #184
vickychijwani wrote: post #124

Earlier quoted context omitted.

I agree it’s not likely , but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s: Example 5: An AI given a goal within a tightly-constrained sandbox figures the…

Example 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable. The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example). I think it’s a poten…

How is bringing down the whole power grid in any particular country an extinction level event? I'm pretty sure that even in the worst case scenario it would be like a month of chaos in one particular part of the world at most, hardly something that would have a long-lasting impact on the humankind's ability to survive at large.

If the answer is "they'd at least try to nuke the country that did it in response", then once again, LLMs are not the main threat.

Re: I resigned from Anthropic today

#749

Post by higeorge13 »

Anyone fearing that all this marketing BS and that race to AGI (?) will destroy the SaaS industry (and others?) and will also kill a few millions of jobs worldwide? I think the US has invested in an AI battle vs China and protect the currently all-in-AI stock market at all costs, and nobody has thought of what will happen to normal people with regular jobs in the tech (or not) industry.

Re: I resigned from Anthropic today

#750

Post by throwaway827367 »

This whole "ASI is going to destroy humanity so we must build it before others do" reminds me of a convo I had with my friend many years ago. He was an officer in the state security service in the dictatorship we both lived in at the time. He said something along these lines: "We had a chat with my colleagues about how are serving the evil. But we decided it's better if this insitution is staffed with decent people."

History did put their theory to test after all. It didn't work.

Post reply on HN