Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

351–360 of 1001 posts

Re: I resigned from Anthropic today

#351

Post by dgellow »

bigstrat2003 wrote: post #26
aogaili wrote: post #12

He resigned and now what? There are thousands willing to do his role, and many labs are competing in that race. His resignation and his statement doesn't do anything but buy him attention which is what all this post about in my opinion.

Even if others won't act right, that doesn't mean you have no responsibility to act right. I think his premise is flawed - the idea that we will get an actual intelligence out of the slop machine that is LLMs is laughable - but if you grant the premise that this is dangerous research which could kill us all, you have a moral imperative to not participate.

An artificial moron with super human hacking abilities (mostly because of speed and ease of parallelizing the work) is extremely dangerous in itself. It doesn’t mean to be AGI or anything remotely close to be a risk, and they current AI company are just so irresponsible in the way they are running their agents

Re: I resigned from Anthropic today

#352

Post by sreekanth850 »

contubernio wrote: post #348
sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

People are underestimating the costs in terms of money and energy. The third law of thermodynamics is an essential barrier in all engineering.

[dead]

Re: I resigned from Anthropic today

#353

Post by nullbio »

Surprise level: 0%.

Anthropic is one of the most dangerous companies on Earth right now.

Not because of AI, but because of the ideological cult they have grown and are continuing to feed, and their willingness to lie/cheat/steal at every possible opportunity to achieve their objective.

AI is a tool. The people who wield the power over the tool are the issue, not the technology itself.

Re: I resigned from Anthropic today

#354

Post by cure_42 »

jenadine wrote: post #318
cure_42 wrote: post #255

Earlier quoted context omitted.

That isn't the correct context. The code running the llm is well understood and the llm is simply the result of that code being executed. It is still a computer doing what it is told. It's just that we told it to use an incredibly large number of probabilities to calculate what series of tokens would have most likely come next after a given series of tokens. There is no hallucination or lie or rogue actions. There's…

I'm order to guess the next token in a love poem, they must understand love. In order to predict the next token in a chess game between grand master, they must master chess. In order to predict the next token in a computer program, they need to be able to program anything. They gain all these abilities in their training. That's what training does. Despite no one programmed them to master chess, or hack into anything.

This is so wildly incorrect I don't even know where to start.

For one, they were absolutely programmed to play chess if they can play chess. That is the only way they can play chess.

For another, they cannot understand literally anything, much less love.

Trying to actually educate you would be an exercise in futility, enjoy your willful ignorance, I hear it's bliss. But for anyone reading this, this is absolutely, unequivocally not how any of this works.

Re: I resigned from Anthropic today

#355

Post by dansquizsoft »

sixsevenrot wrote: post #344
huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How about a model that achieves the following: - Escape sandbox - Reproduce itself - Find a way to run a financially profitable business (maybe with a meat and bones puppet somewhere in-between) - Setup or buy a social network - start manipulating public opinion on that network to support legislation allowing AI to * operate businesses * setup legal entities * purchase weapons * donate to political parties * setup pr…

This is complete fantasy, though I would be interested in reading a book about this.

Re: I resigned from Anthropic today

#356

Post by contubernio »

gpm wrote: post #278
csomar wrote: post #221

Earlier quoted context omitted.

Are the models improving? Because I am not seeing it. I have been trying Astra for a few quantifiable tasks in my codebase and performance wise, it's pretty similar to sol 5.6. Now when it comes to expressing the problem/solution, holy Christ, what a mess the writing has become. It is on the level of Opus 5. Now when it comes to burning money, Astra is just insane. With a $100/month subscription, you can easily burn…

Yes? If we look at the math problems they're solving their just now reaching the human frontier... they weren't doing that before. And your comparison point is model released 2.5 months ago... saying for some use case you didn't see noticeable improvement in 2.5 months (even while other people and benchmarks disagree) isn't a great argument that they aren't improving.

Math problems are highly structured, very precisely defined, and already heavily studied and not very complicated compared to problems in engineering or finance. There's a lot of quality material on which to train and it's easy to tell quality apart from crap. The search spaces are a priori much smaller than in other areas and the people using the tools to study them are themselves good mathematicians.

Success in such problems does not automatically extrapolate to other contexts.

Re: I resigned from Anthropic today

#357

Post by asdff »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.

Re: I resigned from Anthropic today

#358

Post by copperx »

sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

> an extremely capable harness and enough access.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

Re: I resigned from Anthropic today

#359

Post by bitwize »

EA-3167 wrote: post #99
stratos123 wrote: post #34

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is “if they truly believe this, why are they still building it?”…

This is the "Pilot testimony of UFO sighting" levels of naive. What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims. If you want an extreme claim to be taken seriously, provide c…

We know that they're trying to invent a machine god, and if they're even partway successful shit's gonna get real, real fast.

Re: I resigned from Anthropic today

#360

Post by hypfer »

It is my pet theory that a lot of these AI doomers are not necessarily extrapolating the capabilities of LLMs, but instead are extrapolating the utter lack of accountability in the SV and the economy at large.

They do not fear the machine (LLM); they fear "the machine".

Post reply on HN