Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

361–370 of 1001 posts

Re: I resigned from Anthropic today

#361

Post by rob74 »

sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it?

OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.

Re: I resigned from Anthropic today

#362

Post by sreekanth850 »

rob74 wrote: post #361
sreekanth850 wrote: post #345

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it? OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc.")…

D o you think a machine gun is marter than humans? or a car is smarter than Human brain? Human doesnt need to test, if the outcome can be tested deterministically by harness. The model tries. The harness checks whether the expected outcome happened. If not, retry.

Re: I resigned from Anthropic today

#364

Post by deaux »

platinumrad wrote: post #108
x312 wrote: post #96

This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity. This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human compr…

> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right con…

> write an English paragraph that doesn't make me want to claw my eyes out.

LLMs are very much capable of that. Your belief in the opposite has two causes. Firstly the toupee fallacy. You don't notice LLM written text that doesn't make you claw your eyes out. Second is defaults. The huge majority of people who writes text with LLMs just uses default Claude/GPT models, and put near zero effort in making it sound human. Those models are indeed bad at it by default so they need a lot of effort to overcome it. In cases like Opus 5 it's near impossible to overcome. That doesn't generalize to "LLMs".

Re: I resigned from Anthropic today

#365

Post by dgellow »

aogaili wrote: post #122

HN crowed need to make up their minds.. Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots? Are they proofing or stealing math?

It’s a discussion forum so you will of course see different perspectives. There isn’t an HN mind, and it’s not as simple as you make it seem. We don’t need a god to damage humanity significantly, an artificial moron can be as dangerous as an AI god if it is given super-human capabilities, similar to what OpenAI did for the hugging face hack (which wasn’t at all caused by a rogue agent)

Re: I resigned from Anthropic today

#366

Post by jenadine »

epihelix wrote: post #301
platinumrad wrote: post #97

Earlier quoted context omitted.

Anthropic is a company full of basilisk believers.

Yes, but the really weird thing is that they seem to: a) believe that what they're creating is a basilisk, and b) keep trying harder to do this while staring right at it I think they're very deluded about (a) -- but if they do actually believe this (and it really seems like a decent proportion of Anthropic truly does), then why keep doing (b)? That seems to be why this individual resigned, but I'm surprised it's not…

He was referring to this basilisk https://en.wikipedia.org/wiki/Roko%27s_basilisk

In short, this is the believe that a god-like AI could punish them retroactively, for not having done all that was in their power to create this AI.

(A bit similar to some religious believe that a god could punish you after your death if you did not spend your live "pleasing" said god during your life)

Re: I resigned from Anthropic today

#367

Post by minamete »

aogaili wrote: post #256
dtdynasty wrote: post #237

Earlier quoted context omitted.

Don't you think it's a good thing that hacker news isn't a monolith on their beliefs?

Of course it is good. I'm just pointing out how large the gap in narrative is. On one hand, we have people quitting their job believing AI will end humanity in few years. And on the other hand, we have people believing that this tech is nothing more than a statistical tool stealing from others and it can't be trusted with anything. Both views can't be true.

Then it’s not a narrative and you have different people believing different things and adding to the discussion. I think this is a good thing.

Re: I resigned from Anthropic today

#368

Post by asdff »

riffraff wrote: post #321
foogazi wrote: post #217

Earlier quoted context omitted.

> I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. Nuclear weapons don’t have AI but AI can have nuclear weapons

Abstractly, yes but concretely, how? Many terrorist organizations would like to have a nuclear bomb, but don't.

"Department of War has announced a new partnership with blablalbablalba-AI..."

World ends shortly thereafter.

Re: I resigned from Anthropic today

#369

Post by altcognito »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

> Non-example 2: There are worries about AI-facilitated biological weapons. I haven’t seen any evidence that’s happening.

I think this is a good example of poor risk management reasoning. there is evidence bioengineering is already happening. No, nobody is going to announce when somebody has decided to use these tools (even if isn’t an LLM) to bioengineer a weapon. Are the tools power enough to do so? Not sure.

But I’m just ambivalent. It’s probably bad. But there’s nothing to do about it. We’ve really only just pulled back the lid on Pandora’s box.

Re: I resigned from Anthropic today

#370

Post by nvdc »

stratos123 wrote: post #34

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is “if they truly believe this, why are they still building it?”…

i don't think everything that comes out like this is marketing. however, i do think that these companies are largely staffed by "true believers" (anthropic especially) -- people who are so lost in the sauce and embedded in very specific, very peculiar, sf-based rationalist circles where the ai apocalypse is a foregone conclusion.

i understand that these models are powerful and pose certain risks. i use them daily for work and the pace of improvement has been pretty remarkable. that said, i don't buy for a second the borderline-religious proclamations coming from some of these researchers, even if i believe that they are making these claims in earnest

Post reply on HN