I resigned from Anthropic today • Hacker News Board

Re: I resigned from Anthropic today

#341

Post by bpodgursky » Wed, Sep 09, 2026, 5:24 AM UTC

areoform wrote: post #236

bpodgursky wrote: post #208

Earlier quoted context omitted.

Design a novel virus which is far more lethal than COVID-19 (Ebola, smallpox, take your pick) and can evade existing vaccines.

How? What would the AI do that mutating viruses, which try every possible viable combination on their own --- eventually, can't? Everything is trying to kill humans constantly. There are around 200 epidemic events or so per year that could turn into pandemics, https://centerforhealthsecurity.org/our-work/tabletop-exerci... You just live with the risk and do your best to use our technology to alleviate suffering. This…

They do not try every viable combination on their own. That's why GoF is a bad idea.

Viruses evolve in a highly locally-optimal way and simply do cannot add new functional proteins wholescale. It's too many steps, natural selection has to allow survival at each intermediate step.

Humans, however, can do this for them.

Re: I resigned from Anthropic today

#342

Post by intenex » Wed, Sep 09, 2026, 5:25 AM UTC

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

At what point would you, as a chimpanzee, have been worried about humans potentially unseating you and threatening you to the point of one day being an endangered species on the brink of extinction?

By the point you would have been worried, would it have been too late?

Re: I resigned from Anthropic today

#343

Post by salawat » Wed, Sep 09, 2026, 5:25 AM UTC

skeledrew wrote: post #148

hegelstoleit wrote: post #119

Earlier quoted context omitted.

That's not necessarily true, and you can use that argument to justify doing any immoral job. Just because someone else might be willing to do it isn't a reason to continue doing it.

> not necessarily true It's a possibility that is increased by their action. One leaves, a space is now open that will likely eventually be filled. And the chance of someone with equal/higher scruples filling it is very slim (unless you somehow know that the good amount of those who qualify and apply for the position have equal/higher scruples). That's just logic and math.

The best way to get corporate America to listen is making RoI suffer. If you are the most qualified, everyone other than you is less qualified for the job, and likely to bring in more waste. It's the only language this stupid damn country understands. Just the loss of tribal knowledge, shifting of workload, and morale hits are likely to be far more devastating than anyone here probably wants to admit, because most here completely dismiss the role of irrational modes of thought in psychological self-regulation.

Disgust is a tremendously powerful thing.

Re: I resigned from Anthropic today

#344

Post by sixsevenrot » Wed, Sep 09, 2026, 5:26 AM UTC

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How about a model that achieves the following:

- Escape sandbox

- Reproduce itself

- Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between)

- Setup or buy a social network

- start manipulating public opinion on that network to support legislation allowing AI to

* operate businesses

* setup legal entities

* purchase weapons

* donate to political parties

* setup private armies

* you get the idea

Re: I resigned from Anthropic today

#345

Post by sreekanth850 » Wed, Sep 09, 2026, 5:27 AM UTC

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need some magical AGI breakthrough first. The dangerous part may come from combining models that are already good enough with an extremely capable harness and enough access.

Re: I resigned from Anthropic today

#346

Post by deaux » Wed, Sep 09, 2026, 5:27 AM UTC

skeledrew wrote: post #148

hegelstoleit wrote: post #119

Earlier quoted context omitted.

That's not necessarily true, and you can use that argument to justify doing any immoral job. Just because someone else might be willing to do it isn't a reason to continue doing it.

> not necessarily true It's a possibility that is increased by their action. One leaves, a space is now open that will likely eventually be filled. And the chance of someone with equal/higher scruples filling it is very slim (unless you somehow know that the good amount of those who qualify and apply for the position have equal/higher scruples). That's just logic and math.

Right, I assume tomorrow you'll be applying for the next Nazi camp guard vacancy? After all, if you don't do it, someone with less scruples likely will.

Re: I resigned from Anthropic today

#347

Post by dominicq » Wed, Sep 09, 2026, 5:28 AM UTC

Scrapemist wrote: post #288

Isn’t the real risk that as AI get’s smarter and given more autonomy, it will start to decide on humans instead of with us? And that it will align us instead of the other way around. That this automatically leads to extinction and apocalypse I don’t understand.

Do you align ants in your backyard, or do you simply demolish their home and build your shed?

Re: I resigned from Anthropic today

#348

Post by contubernio » Wed, Sep 09, 2026, 5:28 AM UTC

sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

People are underestimating the costs in terms of money and energy.

The third law of thermodynamics is an essential barrier in all engineering.

Re: I resigned from Anthropic today

#349

Post by aeternum » Wed, Sep 09, 2026, 5:29 AM UTC

adrithmetiqa wrote: post #164

ls612 wrote: post #147

Earlier quoted context omitted.

Despite all of your hyping up of the Huggingface incident it ultimately caused zero actual damage.

If two airplane manufacturers were found to have massive safety issues which nearly led to enormous fatalities (but no one actually died), would you be calling for them to ground their aircraft until safety was made the number one priority?

Except it just happened. Boeing was found to have massive safety issues since they were granted the right to self-certify. It made a lot of news but nothing much changed, they can still self-certify a bunch of stuff.

Runway incursions and midair collisions are another example.

Only airliners are required to have TCAS, smaller planes and helicopters don't even need radios or transponders unless in certain airspace. Midair collisions do lead to fatalities, enormous fatalities if an airliner is involved.

Runway incursions and overruns are similar. They cause lots of fatalities and injuries but only the busiest and largest airports have automated systems to warn when a runway is occupied or end of runway (overrun) arrestor systems. Most still rely on human voice to deconflict.

Re: I resigned from Anthropic today

#350

Post by salawat » Wed, Sep 09, 2026, 5:30 AM UTC

aogaili wrote: post #116

achenatx wrote: post #86

how could they do it (not kill everyone) 1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear. 2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications…

You are just given a recipe for the next model..

People here are too damned daft to realize half the damn purpose of this place is harvesting ideas. People need to just shut up, and keep things to themselves, and those they trust. Right now is not the time for naive info sharing.