Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Changes in the system prompt between Claude Opus 4.6 and 4.7

simonwillison.net

Post reply on HN
Print view View original

111–120 of 240 posts

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#111

Post by alwillis »

MoltenMan wrote: post #88
zozbot234 wrote: post #79

Earlier quoted context omitted.

That part of the system prompt is just stating that telling someone who has an actual eating disorder to start counting calories or micro-manage their eating in other ways (a suggestion that the model might well give to an average person for the sake of clear argument, which would then be understood sensibly and taken with a grain of salt) is likely to make them worse off, not better off. This seems like a common-sen…

The problem is that this is an incredibly niche / small issue (i.e. At some point you just have to accept that llm's, like people, make mistakes, and that's ok!

>The problem is that this is an incredibly niche / small issue (i.e. It's not a niche issue at all. 29 million people in the US are struggling with an eating disorder [1].

> This single paragraph is going to legitimately cost anthropic at least 4, maybe 5 digits.

It's 59 out of 3,791 words total in the system prompt. That's 1.48%. Relax.

It should go without saying, but Anthropic has the usage data; they must be seeing a significant increase in the number of times eating disorders come up in conversations with Claude. I'm sure Anthropic takes what goes into the system prompt very seriously.

[1]: from https://www.southdenvertherapy.com/blog/eating-disorder-stat...

The trajectory is troubling. Eating disorder prevalence has more than doubled globally since 2000, with a 124% increase according to World Health Organization data. The United States has seen similar trends, with hospitalization rates climbing steadily year over year.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#112

Post by jug »

embedding-shape wrote: post #28
an0malous wrote: post #16

Earlier quoted context omitted.

I’m just surprised this works at all. When I was building AI automations for a startup in January, even 1,000 word system prompts would cause the model to start losing track of some of the rules. You could even have something simple like “never do X” and it would still sometimes do X.

Two things; the model and runtime matters a lot, smaller/quantized models are basically useless at strict instruction following, compared to SOTA models. The second thing is that "never do X" doesn't work that well, if you want it to "never do X" you need to adjust the harness and/or steer it with "positive prompting" instead. Don't do "Never use uppercase" but instead do "Always use lowercase only", as a silly examp…

It's interesting to note here that Anthropic indeed don't use "do not X" in the Opus system prompts. However, "Claude does not X" is very common.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#113

Post by sensanaty »

cfcf14 wrote: post #4

I'm curious as to why 4.7 seems obsessed with avoiding any actions that could help the user create or enhance malware. The system prompts seem similar on the matter, so I wonder if this is an early attempt by Anthropic to use steering vector injection? The malware paranoia is so strong that my company has had to temporarily block use of 4.7 on our IDE of choice, as the model was behaving in a concerningly unaligned w…

Their marketing is going overtime into selling the image that their models are capable of creating uber sophisticated malware, so every single thing they do from here on out is going to have this fear mongering built in.

Every statement they make, hell even the models themselves are going to be doing this theater of "Ooooh scary uber h4xx0r AI, you can only beat it if you use our Super Giga Pro 40x Plan!!". In a month or two they'll move onto some other thing as they always do.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#114

Post by saagarjha »

akdor1154 wrote: post #89
saagarjha wrote: post #78

Earlier quoted context omitted.

That’s because Anthropic does not consider their model as having personality but rather that it simulates the experience of an abstract entity named Claude.

That sounds really interesting, but my google-fu is not up to task here, I'm getting pages and pages of nonsense asking if Claude is conscious. Can you elaborate?

I actually think this is pretty straightforward if you think of it something like

  class Claude {}
  
  Claude anthropicInstance = new Claude();
  anthropicInstance.greet();
Just like a "Cat" object in Java is supposed to behave like a cat, but is not a cat, and there is no way for Cat@439f5b3d to "be" a cat. However, it is supposed to act like a cat. When Anthropic spins up a model and "runs" it they are asking the matrix multipliers to simulate the concept of a person named Claude. It is not conscious, but it is supposed to simulate a person who is conscious. At least that is how they view it, anyway.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#116

Post by dymk »

teaearlgraycold wrote: post #102
jeffrwells wrote: post #66

Earlier quoted context omitted.

Another way to think about it: every single user of Claude is paying an extra tax in every single request

Well the system prompt is probably permanently cached.

Takes up a portion of the context window, though

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#117

Post by ChadNauseam »

fineIllregister wrote: post #104
bojan wrote: post #73

Earlier quoted context omitted.

There are many, many companies out there putting out products that cause cancer. Think about alcohol, tobacco, internal combustion engines, just to name a few most obvious examples.

> alcohol, tobacco, internal combustion engine Yes, the companies providing these products are sued a lot and are heavily regulated, too.

If you get cancer from drinking alcohol, smoking cigarettes or breathing particles emitted by ICE engines in their standard course of operation, you generally can't sue the manufacturer.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#118

Post by wongarsu »

teaearlgraycold wrote: post #102
jeffrwells wrote: post #66

Earlier quoted context omitted.

Another way to think about it: every single user of Claude is paying an extra tax in every single request

Well the system prompt is probably permanently cached.

On API pricing you still pay 10% of the input token price on cache reads. Not sure if the subscription limits count this though.

And of course all conversations now have to compact 80 tokens earlier, and are marginally worse (since results get worse the more stuff is in the context)

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#119

Post by rcfox »

l5870uoo9y wrote: post #74
walthamstow wrote: post #24

The eating disorder section is kind of crazy. Are we going to incrementally add sections for every 'bad' human behaviour as time goes on?

Could be that Claude has particular controversial opinions on eating disorders.

There are communities of people who publicly blog about their eating disorders. I wouldn't be surprised if the laymen's discourse is over-represented in the LLM's training data compared to the scientific papers.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#120

Post by slashdave »

ceejayoz wrote: post #15
varispeed wrote: post #7

Before Opus 4.7, the 4.6 became very much unusable as it has been flagging normal data analysis scripts it wrote itself as cyber security risk. Got several sessions blocked and was unable to finish research with it and had to switch to GPT-5.4 which has its own problems, but at least is not eager to interfere in legitimate work. edit: to be fair Anthropic should be giving money back for sessions terminated this way.

> edit: to be fair Anthropic should be giving money back for sessions terminated this way. I asked it for one and it told me to file a Github issue. Which I interpreted as "fuck off".

You asked the agent directly for a refund?
Post reply on HN