Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I resigned from Anthropic today

twitter.com

Post reply on HN
Print view View original

361–370 of 985 posts

Re: I resigned from Anthropic today

#361

Post by asdff »

huitzitziltzin wrote: post #94

“ No other human activity poses this level of danger.” I really, really disagree with that statement. I don’t think ai models come close to nuclear weapons or to run-of-the-mill, everyday carbon emissions in terms of danger to humanity. What’s the most dangerous thing that’s happened with an LLM so far? (This question is serious - maybe I don’t know the right examples.) Example 1: I’m aware of a small number of peopl…

How long until kegsbreth hooks the nuclear weapon system into some insider traded black box llm company we hope doesn't end civilization from incompetence or malice? I mean just look where things are going and the sort of people who are steering the damn ship.

Re: I resigned from Anthropic today

#362

Post by copperx »

sreekanth850 wrote: post #349

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

> an extremely capable harness and enough access.

Give enough access to a fuzzer and it's exactly as dangerous as an LLM. LLMs don't even have a moat in this domain.

Re: I resigned from Anthropic today

#363

Post by bitwize »

EA-3167 wrote: post #99
stratos123 wrote: post #34

The people building AI earnestly believe that it could kill us all by the end of the decade. This is not a marketing stunt. If anything, many executives and senior researchers will couch their phrasing in the press to sound sensible - but I hear the same people express fear privately. No other human activity poses this level of danger. A common response is “if they truly believe this, why are they still building it?”…

This is the "Pilot testimony of UFO sighting" levels of naive. What's more likely? Anthropic is doing some deeply unethical marketing in the lead up to their multi-trillion dollar IPO? Or they're inventing a machine god? There's ample evidence of the former because that's their entire business model, but no evidence whatsoever to support the latter claims. If you want an extreme claim to be taken seriously, provide c…

We know that they're trying to invent a machine god, and if they're even partway successful shit's gonna get real, real fast.

Re: I resigned from Anthropic today

#364

Post by hypfer »

It is my pet theory that a lot of these AI doomers are not necessarily extrapolating the capabilities of LLMs, but instead are extrapolating the utter lack of accountability in the SV and the economy at large.

They do not fear the machine (LLM); they fear "the machine".

Re: I resigned from Anthropic today

#365

Post by rob74 »

sreekanth850 wrote: post #349

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it?

OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc."), that's of course more suitable for a harness.

Re: I resigned from Anthropic today

#366

Post by sreekanth850 »

rob74 wrote: post #365
sreekanth850 wrote: post #349

I think people here still evaluating the model in isolation. It is the combination that matters, model + strong harness + tools + long running autonomy + memory + retries + parallel agents + code execution + credentials + access to real systems. The model does not need to be perfect. If it fails 30% of the time, the harness can retry, verify, branch, use another agent and keep going. I don't think we necessarily need…

Doesn't this just move the need to be smarter from the model to the harness - if a human sometimes can't tell whether a model has produced something correct or just mostly correct-looking BS, how can an automated harness do it? OTOH, if the goal is simple ("break into a protected system") rather than more complex ("write an application that satisfies all requirements on all supported devices/screen resolutions etc.")…

D o you think a machine gun is marter than humans? or a car is smarter than Human brain? Human doesnt need to test, if the outcome can be tested deterministically by harness. The model tries. The harness checks whether the expected outcome happened. If not, retry.

Re: I resigned from Anthropic today

#368

Post by deaux »

platinumrad wrote: post #108
x312 wrote: post #96

This is increasingly the consensus I see also on the academic side of AI/safety research. Specifically that AI poses an existential risk to humanity. This was a fringe belief until recently, but the progress of AI in research is impossible to ignore. Epecially in math, where not only has AI outstripped humans in generative ability, but is able to create scientific knowledge which is beyond the capacity of human compr…

> There's clearly no intelligence task that AIs can't do due to some magic fundamental constraint. Yes there is: write an English paragraph that doesn't make me want to claw my eyes out. LLMs are not better than human mathematicians (or security researchers) in all respects, just some specific ways (e.g. not having to take a lunch break) that make them good at exhaustively searching for an answer, given the right con…

> write an English paragraph that doesn't make me want to claw my eyes out.

LLMs are very much capable of that. Your belief in the opposite has two causes. Firstly the toupee fallacy. You don't notice LLM written text that doesn't make you claw your eyes out. Second is defaults. The huge majority of people who writes text with LLMs just uses default Claude/GPT models, and put near zero effort in making it sound human. Those models are indeed bad at it by default so they need a lot of effort to overcome it. In cases like Opus 5 it's near impossible to overcome. That doesn't generalize to "LLMs".

Re: I resigned from Anthropic today

#369

Post by dgellow »

aogaili wrote: post #122

HN crowed need to make up their minds.. Are LLMs about to be a god that will annihilate humanity? Or are they statistical parrots? Are they proofing or stealing math?

It’s a discussion forum so you will of course see different perspectives. There isn’t an HN mind, and it’s not as simple as you make it seem. We don’t need a god to damage humanity significantly, an artificial moron can be as dangerous as an AI god if it is given super-human capabilities, similar to what OpenAI did for the hugging face hack (which wasn’t at all caused by a rogue agent)

Re: I resigned from Anthropic today

#370

Post by cure_42 »

doix wrote: post #313
cameldrv wrote: post #293

Earlier quoted context omitted.

You’re just a bunch of molecules following the laws of physics. It’s all just physics and chemistry, and those are well understood. Now explain the causes of World War I using chemistry and physics. Simple, right?

> It’s all just physics and chemistry, and those are well understood. Not really. We cannot model physics and chemistry to a level which allows us to accurately predict a humans action (even a tiny time-step into the future) This is vastly different to an LLM, where the model is the model (for a lack of better phrasing).

To be fair, we can't model human language well enough to accurately predict what an actual human will say either. Our ability to accurately model physics is similar to our ability to accurately model human language. And we make immense use of both kinds of model, despite their flaws, inaccuracies, and inability to ever be perfect.
Post reply on HN