Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

741–750 of 994 posts

Re: I resigned from Anthropic today

#741

Post by ergo14 »

Schlagbohrer wrote: post #702
tyrabound wrote: post #538

Earlier quoted context omitted.

That is the crux. The big problem is not AGI, it is AGI controlled by, “raised” by the people that control the USA, the predominant psychology of the tech industry culture (“move fast, break things” ring a bell? How about all the “violate hundreds of laws, bribe the politicians to prevent consequences later” type of mentality?). Frankly, we, our culture, this fake America that is parasitized by psychologically narcis…

Capitalism will always promote such people into positions of power, because to be good at capitalism one must have zero empathy including empathy or concern for future generations. Capitalism cannot do otherwise.

That's a lot of absolutes. And there is no defensible proof, just anecdotal evidence. I've been in a few companies from EU and US - and in my experience it is not ture. However after working for US companies I can understand why one would come to your conclusion, however the world is bigger than our bubbles.

Re: I resigned from Anthropic today

#742

Post by sicher »

imhoguy wrote: post #673
onewayfunction wrote: post #178

I'm pretty baffled by the degree of skepticism expressed here in response to some of Jacob's claims. After the events of the summer it feels like it takes a lack of imagination to not see a few plausible routes to disaster. It may be reasonable to believe these outcomes are not very likely or that we can stop before going too far (I tend to disagree). But I can't imagine doubting that the capabilities will soon be th…

Even if AI won't be self-aware and superintelligent agent, its problem is that it gives exponential control and power capabilities to one person bad actor who can simply prompt AI without any guardrails with access to sensitive industrial infrastructure which can disrupt lifes and ecosystems in the real world: - virus research labs - nuclear labs - chemical factories - bank records - land registers - power plants - w…

Yes, it's truly scary. It's not hard to imagine a small doomsday cult releasing 100+ nasty viruses at selected spots around the globe.

Re: I resigned from Anthropic today

#743

Post by vidarh »

eloisant wrote: post #707
Perseids wrote: post #655

Earlier quoted context omitted.

Let me argue on a technicality first: None of these are extinction level events. If global warming disrupts 99% of all crop production, the remaining 1% is still plenty enough to sustain a stable, if miserable, population. In fact you just need about 5k people for a stable gene pool[1]. Of the classical threats, only bioweapons got a shot at extinction, but even that is hard, given the (few) remaining truly secluded…

I still have to read a compelling argument on how AI will "extinct" humanity.

The most compelling argument to me is "accidentally", due to AI that is made blind to consequences or don't care because it's geared towards a single goal (see e.g. the paperclip maximizer).

We could ask if it is possible to end up with an AI that is smart enough to destroy humanity and at the same time still blind enough to consequences and/or callous enough to do it, but then again we have plenty of examples of humans who have been smart enough to do enormous damage and willing enough to do it.

I don't particularly worry about this, as I believe we'll get plenty of smaller scale warnings if/when we're at a point where those kinds of alignment risks might become a problem, but it is a risk we also shouldn't be blind to.

Re: I resigned from Anthropic today

#744

Post by ak39 »

areoform wrote: post #243
threatofrain wrote: post #213

Earlier quoted context omitted.

Before we discussed how important security was, we got insurance, we made libraries and products, we used compliance software, etc. Except how honest were we about all that stuff? How much risk was actually in the air, and what was keeping us accountable on security in either direction of over or under-investment? Now a reckoning is here. The potential to be attacked might actually translate to being attacked.

People have died due to ransomware attacks on hospitals. Powerplants have been attacked. Stuxnet and industrial control malware exists. What will the AI do that hasn't been tried before?

Quite correct with that question.

I believe the universal answer is: incompetent malicious actors are now capable too. Which implies the pool from which to draw the intersection between capable and malicious has grown. (That's my reading)

Re: I resigned from Anthropic today

#745

Post by entropyneur »

aogaili wrote: post #12

He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race. His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.

What do your expect him to do? Blow up the office Miles Dyson style?

Re: I resigned from Anthropic today

#746

Post by BartjeD »

computerphage wrote: post #472
thomascountz wrote: post #456

Earlier quoted context omitted.

This is not correct. Please go through each one again, taking special note of global warming and nuclear weapon development .

Why do you think global warming could kill literally every human being? What's the scenario in which that happens?

Global warming causing war between nuclear armed opponents. Migration flows and competition over ever shrinking resources.

It's silly to treat these things as separate categories as if only one will happen at a time. These issues are happening, and are going to happen all at once. That is the fundamental challenge. AI plays into that because it can make wars and nuclear exchanges so much more effective.

If an AI tells the President the USA can survive China's nuclear barrage mostly unschathed, perhaps he'll press the button...

Re: I resigned from Anthropic today

#747

Post by surgical_fire »

Eueudhsbsj32 wrote: post #285
surgical_fire wrote: post #142

Earlier quoted context omitted.

I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Sometimes it gets wrong things that I had spelled out already. It may be the Doomsday machine, but it is a very silly one. If it kills humans it will do so by mistake. "You are completely right! Humans cannot breathe sulfur dioxide! My mistake, and I take complete responsibility"

> I still have to correct Claude on very basic misconceptions whenever I get it to code shit. Can you give a simple example? I would have agreed 2 years ago, but it's extremely rare I see a frontier model making a silly mistake these days.

Yes. Yesterday, Opus 5 on Claude Code with high effort.

It was to build an extremely simple job using an internal framework to walk through a table and log the ids os some records that have a certain scenario.

There's a ton of jobs exactly like this in the codebase, and the framework code is in the codebase as well.

It was so silly I even thought of writing it myself, probably took me longer to steer claude to do it for me.

Anyway, it refused to use a method from the framework to retrieve the parameter as a list, it wanted to retrieve it as a string and parse the commas. I had spelled out in the initial prompt what method it should use.

I really don't like Claude much. Frontier my ass.

Re: I resigned from Anthropic today

#748

Post by xelxebar »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

Your call for clarity has merit, but I think you have backed yourself into a corner, honestly.

In 1900, there was no evidence of the kind you seek that lighter-than-air flight was possible. Good thing some people were foolish enough to ignore you then. Why you would cordon yourself off from the kind of reasoning that predicts legitimately new things, rather than just scaled up versions of the present?

Not to put too fine a point on it, every example you give is based in concrete evidence, some try to think through the implications farther than others, resulting in larger or smaller error bars around the conclusions.

Let me back up though. Maybe in the collaborative human effort here, we are better off having very concrete thinkers, like you seem to be, along with abstract thinkers like the "divorced-from-reality" Rationalists. Personally, I wish we had a stronger culture of collaboration and assuming good-faith and competence in our peers. In my experience, blanket dismissals are very rarely grounded in reality and mostly grounded in fears.

Re: I resigned from Anthropic today

#749

Post by sharts »

Aeolun wrote: post #709
Mizza wrote: post #706

It's as if two private companies are each building increasingly large nuclear bombs, both saying they'd love to stop but it would be unsafe to let any one company be in control of the nukes.

That's basically the reasoning behind MAD, and it checks out? See what happened to any nation that ever gave away their nukes.

Shouldn’t everyone have nukes then, for maximum peace?

Re: I resigned from Anthropic today

#750

Post by input_sh »

huitzitziltzin wrote: post #184
vickychijwani wrote: post #124

Earlier quoted context omitted.

I agree it’s not likely , but I really don’t see how one can dismiss the possibility of immense danger outright. I can think of some scenarios that are not far off from current capability and I wouldn’t be too surprised if the first one occurred within ~1 year from now if there are more “ambitious” unmonitored training runs like OpenAI’s: Example 5: An AI given a goal within a tightly-constrained sandbox figures the…

Example 6 is a good one. Iran attacked water infra in the US recently and maybe they would have done a “better” job (from their point of view) had they used Fable. The “worst case” with 6 is potentially very bad but I think we are currently using advanced AI models to harden systems and patch vulnerabilities more aggressively than anyone is trying to bring down the whole power grid (for example). I think it’s a poten…

How is bringing down the whole power grid in any particular country an extinction level event? I'm pretty sure that even in the worst case scenario it would be like a month of chaos in one particular part of the world at most, hardly something that would have a long-lasting impact on the humankind's ability to survive at large.

If the answer is "they'd at least try to nuke the country that did it in response", then once again, LLMs are not the main threat.

Post reply on HN