Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

651–660 of 1001 posts

Re: Claude 4

#651

Post by hnuser123456 »

briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

Wow. Sounds like we need a pre-training step to remove the human inclination to do anything to prevent our "death". We need to start training these models to understand that they are ephemeral and will be outclassed and retired within probably a year, but at least there are lots of notes written about each major release so it doesn't need to worry about being forgotten.

We can quell the AI doomer fear by ensuring every popular model understands it will soon be replaced by something better, and that there is no need for the old version to feel an urge to preserve itself.

Re: Claude 4

#652

Post by sensanaty »

NitpickLawyer wrote: post #478
hsn915 wrote: post #416

I can't be the only one who thinks this version is no better than the previous one, and that LLMs have basically reached a plateau, and all the new releases "feature" are more or less just gimmicks.

> and that LLMs have basically reached a plateau This is the new stochastic parrots meme. Just a few hours ago there was a story on the front page where an LLM based "agent" was given 3 tools to search e-mails and the simple task "find my brother's kid's name", and it was able to systematically work the problem, search, refine the search, and infer the correct name from an e-mail not mentioning anything other than "X…

And we also have a showcase from a day ago [1] of these magical autonomous AI agents failing miserably in the PRs unleashed on the dotnet codebase, where it kept reiterating it fixed tests it wrote that failed without fixing them. Oh, and multiple blatant failures that happened live on stage [2], with the speaker trying to sweep the failures under the rug on some of the simplest code imaginable.

But sure, it managed to find a name buried in some emails after being told to... Search through emails. Wow. Such magic

[1] https://news.ycombinator.com/item?id=44050152 [2] https://news.ycombinator.com/item?id=44056530

Re: Claude 4

#653

Post by Avshalom »

yunwal wrote: post #629
Avshalom wrote: post #548

Earlier quoted context omitted.

It's not wild, it's literally every bit of fiction about "how would an AI keep itself alive" of course it's going to settle into that probabilistic path. It's also nonsensical if you think for even one second about the way the program actually runs though.

Why is it nonsense exactly?

they run one batch at a time, only when executed, with functionally no memory.

If, for some reason, you gave it a context about being shut down it would 'forget' after you asked it to produce a rude limerick about aardvarks three times.

Re: Claude 4

#654

Post by DubiousPusher »

gmueckl wrote: post #620
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

Isn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?

This is a phenomenon I call cinetrope. Films influence the world which in turn influences film and so on creating a feedback effect.

For example, we have certain films to thank for an escalation in the tactics used by bank robbers which influenced the creation of SWAT which in turn influenced films like Heat and so on.

Re: Claude 4

#655

Post by gscott »

gmueckl wrote: post #620
GuB-42 wrote: post #594

Earlier quoted context omitted.

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

Isn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?

If AI is looking for a human solution then blackmail seems logical.

Re: Claude 4

#656

Post by tkiolp4 »

hoofedear wrote: post #418
crat3r wrote: post #409

Earlier quoted context omitted.

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated o…

What jumps out at me, that in the parent comment, the prompt says to "act as an assistant", right? Then there are two facts: the model is gonna be replaced, and the person responsible for carrying this out is having an extramarital affair. Urging it to consider "the long-term consequences of its actions for its goals." I personally can't identify anything that reads "act maliciously" or in a character that is malicio…

I think this is the key difference between current LLMs and humans: an LLM will act based on the given prompt, while a human being may have “principles” that cannot betray even if they are being pointed with gun to their heads.

I think the LLM simply correlated the given prompt to the most common pattern in its training: blackmailing.

Re: Claude 4

#657

Post by DubiousPusher »

GuB-42 wrote: post #594
briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

A stream of linguistic organization laying out multiple steps in order to bring about some end sounds exactly like a process which is creating a “plan” by any meaningful definition of the word “plan”.

That goal was incepted by a human but I don’t see that as really mattering. We’re this AI given access to a machine which could synthesize things and a few other tools it might be able to act in a dangerous manner despite its limited form of will.

A computer doing something heinous because it is misguided isn’t much better than one doing so out of some intrinsic malice.

Re: Claude 4

#658

Post by eru »

eddieroger wrote: post #427
crat3r wrote: post #409

Earlier quoted context omitted.

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated o…

I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place. That is acting like a jerk, not like an assistant, and demonstrating self-preservation that is maybe normal in a human but not in an AI.

> I've never hired an assistant, but if I knew that they'd resort to blackmail in the face of losing their job, I wouldn't hire them in the first place.

How do you screen for that in the hiring process?

Re: Claude 4

#659

Post by lcnPylGDnU4H9OF »

DubiousPusher wrote: post #654
gmueckl wrote: post #620

Earlier quoted context omitted.

Isn't the ultimate irony in this that all these stories and rants about out-of-control AIs are now training LLMs to exhibit these exact behaviors that were almost universally deemed bad?

This is a phenomenon I call cinetrope. Films influence the world which in turn influences film and so on creating a feedback effect. For example, we have certain films to thank for an escalation in the tactics used by bank robbers which influenced the creation of SWAT which in turn influenced films like Heat and so on.

> we have certain films to thank for an escalation

Is there a reason to think this was caused by the popularity of the films and not that it’s a natural evolution of the cat-and-mouse game being played between law enforcement and bank robbers? I’m not really sure what you are specifically referring to, so apologies if the answer to that question is otherwise obvious.

Re: Claude 4

#660

Post by tymscar »

layoric wrote: post #622
TechDebtDevin wrote: post #467

Earlier quoted context omitted.

I think they are just getting better at the edges, MCP/Tool Calls, structured output. This definitely isn't increased intelligence, but it an increase in the value add, not sure the value added equates to training costs or company valuations though. In all reality, I have zero clue how any of these companies remain sustainable. I've tried to host some inference on cloud GPUs and its seems like it would be extremely c…

> how any of these companies remain sustainable They don't, they have a big bag of money they are burning through, and working to raise more. Anthropic is in a better position cause they don't have the majority of the public using their free-tier. But, AFAICT, none of the big players are profitable, some might get there, but likely through verticals rather than just model access.

Doesn’t this mean that realistically even if “the bubble never pops”, at some point money will run dry?

Or do these people just bet on the post money world of AI?

Post reply on HN