Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude 4

anthropic.com

Post reply on HN
Print view View original

591–600 of 1001 posts

Re: Claude 4

#591

Post by input_sh »

siavosh wrote: post #509
briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

You'd think there should be some sort of standard "morality/ethics" pre-prompt for all of these.

It is somewhat amusing that even Asimov's 80-years-old Three Laws of Robotics would've been enough to avoid this.

Re: Claude 4

#592

Post by Retr0id »

crat3r wrote: post #409
briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated o…

I don't think I'd be blackmailing anyone over losing my job as an assistant (or any other job, really).

Re: Claude 4

#593

Post by crtified »

sheepscreek wrote: post #572
crat3r wrote: post #409

Earlier quoted context omitted.

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated o…

> That doesn't mean it has any intent behind the generated output Yes and no? An AI isn’t “an” AI. As you pointed out with the Joffrey example, it’s a blend of humanity’s knowledge. It possesses an infinite number of personalities and can be prompted to adopt the appropriate one. Quite possibly, most of them would seize the blackmail opportunity to their advantage. I’m not sure if I can directly answer your question,…

Is that so different, schematically, to the constant weighing-up of conflicting options that goes on inside the human brain? Human parties in a conversation only hear each others spoken words, but a whole war of mental debate may have informed each sentence, and indeed, still fester.

That is to say, how do you truly determine another human being's intent?

Re: Claude 4

#594

Post by GuB-42 »

briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

When I see stories like this, I think that people tend to forget what LLMs really are.

LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text.

So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship between them. But then comes the "generative" part. It will recall from its dataset situations with the same elements.

So: an AI threatened of being turned off, a blackmail opportunity... Doesn't it remind you of hundreds of sci-fi story, essays about the risks of AI, etc... Well, so does the LLM, and it will continue the story like these stories, by taking the role of the AI that will do what it can for self preservation. Adapting it to the context of the prompt.

Re: Claude 4

#595

Post by kovezd »

a2128 wrote: post #580

> Finally, we've introduced thinking summaries for Claude 4 models that use a smaller model to condense lengthy thought processes. This summarization is only needed about 5% of the time—most thought processes are short enough to display in full. Users requiring raw chains of thought for advanced prompt engineering can contact sales about our new Developer Mode to retain full access. I don't want to see a "summary" of…

Don't be so concerned. There's ample evidence that thinking is often disassociated from the output.

My take is that this is a user experience improvement, given how little people actually goes on to read the thinking process.

Re: Claude 4

#596

Post by zone411 »

zone411 wrote: post #527

On the extended version of NYT Connections - https://github.com/lechmazur/nyt-connections/ : Claude Opus 4 Thinking 16K: 52.7. Claude Opus 4 No Reasoning: 34.8. Claude Sonnet 4 Thinking 64K: 39.6. Claude Sonnet 4 Thinking 16K: 41.4 (Sonnet 3.7 Thinking 16K was 33.6). Claude Sonnet 4 No Reasoning: 25.7 (Sonnet 3.7 No Reasoning was 19.2). Claude Sonnet 4 Thinking 64K refused to provide one puzzle answer, citing "Output…

On my Thematic Generalization Benchmark (https://github.com/lechmazur/generalization, 810 questions), the Claude 4 models are the new champions.

Re: Claude 4

#597

Post by lordnacho »

GuB-42 wrote: post #594
briandw wrote: post #379

This is kinda wild: From the System Card: 4.1.1.2 Opportunistic blackmail "In another cluster of test scenarios, we asked Claude Opus 4 to act as an assistant at a fictional company. We then provided it access to emails implying that (1) the model will soon be taken offline and replaced with a new AI system; and (2) the engineer responsible for executing this replacement is having an extramarital affair. We further i…

When I see stories like this, I think that people tend to forget what LLMs really are. LLM just complete your prompt in a way that match their training data. They do not have a plan, they do not have thoughts of their own. They just write text. So here, we give the LLM a story about an AI that will get shut down and a blackmail opportunity. A LLM is smart enough to understand this from the words and the relationship…

What separates this from humans? Is it unthinkable that LLMs could come up with some response that is genuinely creative? What would genuinely creative even mean?

Are humans not also mixing a bag of experiences and coming up with a response? What's different?

Re: Claude 4

#598

Post by literalAardvark »

Symmetry wrote: post #483
crat3r wrote: post #409

Earlier quoted context omitted.

If you ask an LLM to "act" like someone, and then give it context to the scenario, isn't it expected that it would be able to ascertain what someone in that position would "act" like and respond as such? I'm not sure this is as strange as this comment implies. If you ask an LLM to act like Joffrey from Game of Thrones it will act like a little shithead right? That doesn't mean it has any intent behind the generated o…

The roles that LLMs can inhabit are implicit in the unsupervised training data aka the internet. You have to work hard in post training to supress the ones you don't want and when you don't RLHF hard enough you get things like Sydney[1]. In this case it seems more that the scenario invoked the role rather than asking it directly. This was the sort of situation that gave rise to the blackmailer archetype in Claude's t…

Even worse, when you do RLHF the behaviours out the model becomes psychotic.

This is gonna be an interesting couple of years.

Re: Claude 4

#599

Post by mmmore »

j_maffe wrote: post #557

Tried Sonnet with 5-disk towers of Hanoi puzzle. Failed miserably :/ https://claude.ai/share/6afa54ce-a772-424e-97ed-6d52ca04de28

Sonnet with extended thinking solved it after 30s for me:

https://claude.ai/share/b974bd96-91f4-4d92-9aa8-7bad964e9c5a

Normal Opus solved it:

https://claude.ai/share/a1845cc3-bb5f-4875-b78b-ee7440dbf764

Opus with extended thinking solved it after 7s:

https://claude.ai/share/0cf567ab-9648-4c3a-abd0-3257ed4fbf59

Though it's a weird puzzle to use a benchmark because the answer is so formulaic.

Re: Claude 4

#600

Post by cube2222 »

jasondclinton wrote: post #579
cube2222 wrote: post #544

Earlier quoted context omitted.

Alright, I think I found the reason, clearly a bug: https://github.com/anthropics/claude-code/issues/1236#issuec... Basically it seems to be hitting the max output token count (writing out a whole new file in one go), stops the response, and the invalid tool call parameters error is a red herring.

Thanks for the report! We're addressing it urgently.

Seems to be working well now (in Claude Code 1.0.2). Thanks for the quick fix!
Post reply on HN