Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: I resigned from Anthropic today

#351

Post by aeternum »

adrithmetiqa wrote: post #164
ls612 wrote: post #147

Earlier quoted context omitted.

Despite all of your hyping up of the Huggingface incident it ultimately caused zero actual damage.

If two airplane manufacturers were found to have massive safety issues which nearly led to enormous fatalities (but no one actually died), would you be calling for them to ground their aircraft until safety was made the number one priority?

Except it just happened. Boeing was found to have massive safety issues since they were granted the right to self-certify. It made a lot of news but nothing much changed, they can still self-certify a bunch of stuff.

Runway incursions and midair collisions are another example.

Only airliners are required to have TCAS, smaller planes and helicopters don't even need radios or transponders unless in certain airspace. Midair collisions do lead to fatalities, enormous fatalities if an airliner is involved.

Runway incursions and overruns are similar. They cause lots of fatalities and injuries but only the busiest and largest airports have automated systems to warn when a runway is occupied or end of runway (overrun) arrestor systems. Most still rely on human voice to deconflict.

Re: I resigned from Anthropic today

#352

Post by salawat »

aogaili wrote: post #116
achenatx wrote: post #86

how could they do it (not kill everyone) 1) rogue state releases a self moving self modifying AI into the wild. It is trained on how to hack, monitor new vulnerability updates, scan code bases to find new vulnerabilities. It constantly replicate and hides in systems so it will be extremely difficult to clear. 2) it hacks into public infrastructure taking down traffic, power, water, air traffic control, communications…

You are just given a recipe for the next model..

People here are too damned daft to realize half the damn purpose of this place is harvesting ideas. People need to just shut up, and keep things to themselves, and those they trust. Right now is not the time for naive info sharing.

Re: I resigned from Anthropic today

#353

Post by dgellow »

bigstrat2003 wrote: post #26
aogaili wrote: post #12

He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race. His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.

Even if others won't act right, that doesn't mean you have no responsibility to act right. I think his premise is flawed - the idea that we will get an actual intelligence out of the slop machine that is LLMs is laughable - but if you grant the premise that this is dangerous research which could kill us all, you have a moral imperative to not participate.

An artificial moron with super human hacking abilities (mostly because of speed and ease of parallelizing the work) is extremely dangerous in itself. It doesn’t mean to be AGI or anything remotely close to be a risk, and they current AI company are just so irresponsible in the way they are running their agents

Re: I resigned from Anthropic today

#354

Post by sreekanth850 »

contubernio wrote: post #350
sreekanth850 wrote: post #347

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

People are underestimating the costs in terms of money and energy. The third law of thermodynamics is an essential barrier in all engineering.

[dead]

Re: I resigned from Anthropic today

#355

Post by nullbio »

Surprise level: 0%.

Anthropic is one of the most dangerous companies on Earth right now.

Not because of AI, but because of the ideological cult they have grown and are continuing to feed, and their willingness to lie/cheat/steal at every possible opportunity to achieve their objective.

AI is a tool. The people who wield the power over the tool are the issue, not the technology itself.

Re: I resigned from Anthropic today

#356

Post by cure_42 »

jenadine wrote: post #320
cure_42 wrote: post #257

Earlier quoted context omitted.

That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's…

I'm order to guess the next token in a love poem, they must understand love. In order to predict the next token in a chess game between grand master, they must master chess. In order to predict the next token in a computer program, they need to be able to program anything. They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.

This is so wildly incorrect I don't even know where to start.

For one, they were absolutely programmed to play chess if they can play chess. That is the only way they can play chess.

For another, they cannot understand literally anything, much less love.

Trying to actually educate you would be an exercise in futility, enjoy your willful ignorance, I hear it's bliss. But for anyone reading this, this is absolutely, unequivocally not how any of this works.

Re: I resigned from Anthropic today

#357

Post by dansquizsoft »

sixsevenrot wrote: post #346
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How about a model that achieves the following: - Escape sandbox - Reproduce itself - Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between) - Setup or buy a social network - start manipulating public opinion on that network to support legislation allowing AI to * operate businesses * setup legal entities * purchase weapons * donate to political parties * setup pr…

This is complete fantasy, though I would be interested in reading a book about this.

Re: I resigned from Anthropic today

#358

Post by contubernio »

gpm wrote: post #280
csomar wrote: post #222

Earlier quoted context omitted.

Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn…

Yes? If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before. And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.

Math problems are highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance. There's a lot of quality material on which to train and it's easy to tell quality apart from crap. The search spaces are a priori much smaller than in other areas and the people using the tools to study them are themselves good mathematicians.

Success in such problems does not automatically extrapolate to other contexts.

Re: I resigned from Anthropic today

#359

Post by asdff »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.

Re: I resigned from Anthropic today

#360

Post by copperx »

sreekanth850 wrote: post #347

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

> an extremely capable harness and enough access.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

Post reply on HN