Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Changes in the system prompt between Claude Opus 4.6 and 4.7

simonwillison.net

Post reply on HN
Print view View original

211–220 of 240 posts

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#211

Post by mpalczewski »

cowlby wrote: post #170

I'm fascinated that Anthropic employees, who are supposed to be the LLM experts, are using tricks like these which go against how LLMs seem to work. Key example for me was the "malware" tool call section that included a snippet with intent "if it's malware, refuse to edit the file". Yet because it appears dozens of times in a convo, eventually the LLM gets confused and will refuse to edit a file that is not malware.…

They aren’t necessarily experts at using Llm’s. They have different incentives as well

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#212

Post by salad-tycoon »

walthamstow wrote: post #24

The eating disorder section is kind of crazy. Are we going to incrementally add sections for every 'bad' human behaviour as time goes on?

They have to secretly add these guardrails on because the alternative would be to train the users out of consulting these things as if they are advanced all-knowing alien-technogawds. And that would be bad for business.

The better solution I think would be a reality/personal responsibility approach, teach the consumers that the burden of interpretation is on them and not the magic 8ball. For example if your AI tells you to kill your parents or that you’ve discovered new math that makes time travel possible, etc then: 1. Stop 2. Unplug 3. Go outside 4. Ask a human for a sanity check.

Since that would be bad for business and take a lot of effort on the user side (while being very embarrassing). Obviously can’t do that right before an IPO & in the middle of global economic war so secretive moral frameworks have to be installed.

If you are what you eat then you believe what you consume. Ironically, I think this undisclosed and hidden moral shaping of billions of people will be the most dangerous. Imagine all the things we could do if we can just, ever-so-slightly, move the Overton window / goal posts on w/e topic day by day, prompt by prompt.

Personally I find AI output insidiously disarming and charming and I think I’m in the norm. So while we’ve been besieged by propaganda since time immemorial I do worry that AI is a special case.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#213

Post by salad-tycoon »

nozzlegear wrote: post #163
ChadNauseam wrote: post #151

Earlier quoted context omitted.

That's just not true. Makes me wondered if you've ever bought a bottle of alcohol before lol. There's no label that says it causes cancer. (Maybe in california because of prop 65?) And I expect cars also have no such labelling, not that it would matter, considering they cause cancer in random passers by who have no opportunity to consent to breathing in auto exhaust or read any labels

> Makes me wondered if you've ever bought a bottle of alcohol before lol. I'm a teetotaler so no, I literally have not. I was mostly thinking about cigarette and tobacco products which are the most glaring, obvious counterpoints. But you'll be happy to learn that virtually all vehicles in the US also come with operating manuals that profusely warn people not to breathe in the exhaust from the vehicle.

Don’t worry, every bottle in the US has the surgeon general’s warning on it and it doesn’t call out cancer, yet. Adding cancer to the ills of booze was proposed in 2025 so your intuition was correct, directionally.

On every bottle:

Alcoholic Beverage Labeling Act of 1988

“ GOVERNMENT WARNING: (1) According to the Surgeon General, women should not drink alcoholic beverages during pregnancy because of the risk of birth defects. (2) Consumption of alcoholic beverages impairs your ability to drive a car or operate machinery, and may cause health problems"

Cancer proposal: https://www.mdanderson.org/cancerwise/not-just-a-hangover--t...

https://www.ttb.gov/regulated-commodities/beverage-alcohol/d...

(As if adding this text will do anything other than reduce the companies liability, rofl)

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#214

Post by salad-tycoon »

nozzlegear wrote: post #143
arcanemachiner wrote: post #93

Earlier quoted context omitted.

Why stop there? We could jam up the system prompt with all kinds of irrelevant guardrails to prevent harm to groups X, Y, and Z!

This but unironically. Preventing harm is good, actually.

Because it dumbs everything down, makes the output quality worse and more expensive, and removes personal agency and is dehumanizing. Plus, does it actually prevent harm, do we have evidence?

Finally, what is often missed is what if an actual good is decided harmful or something that is harmful is decided by AI company board XYZ to be “good”?

I think censorship is bad because of that danger. Quis custodiet ipsos custodes (who will watch the watchers).

Instead of throwing ourselves into that minefield of moral hazard, we should be lifting each other up to the tops of our ability and not infantilizing / secretly propagandizing each other.

Well, ideally at least.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#215

Post by bootlooped »

Lucasoato wrote: post #182
jrvarela56 wrote: post #169

The past month made me realize I needed to make my codebase usable by other agents. I was mainly using Claude Code. I audited the codebase and identified the points where I was coupling to it and made a refactor so that I can use either codex, gemini or claude. Here are a few changes: 1. AGENTS.md by default across the codebase, a script makes sure CLAUDE.md symlink present wherever there's an AGENTS.md file 2. Skill…

Have you got any advice in making agents from different providers work together? In Claude, I’ve seen cases in which spawning subagents from Gemini and Codex would raise strange permission errors (even if they don’t happen with other cli commands!), making claude silently continue impersonating the other agent. Only by thoroughly checking I was able to understand that actually the agent I wanted failed.

I would eliminate the possibility of sandbox conflicts by 1) making sure any subagents are invoked with no sandbox (they should still be covered under the calling agent's sandbox) 2) make sure the calling agent's sandbox allows the subagents access to the directories they need (ex: ~/.gemini, ~/.codex).

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#216

Post by salad-tycoon »

pllbnk wrote: post #196
walthamstow wrote: post #24

The eating disorder section is kind of crazy. Are we going to incrementally add sections for every 'bad' human behaviour as time goes on?

Seems so, unless we manage to pivot to open weight models. Hopefully, Chinese will lead the way along with their consumer hardware. Hard for me to say this because I have always been pro-Western and suddenly it seems like the world has flipped.

I feel the same way for a while now but especially recently. It’s been obvious for a while I suppose but greatly clarified recently.

I have just one question for you pllbnk, are we the baddies?

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#217

Post by j-bos »

stingraycharles wrote: post #133
j-bos wrote: post #131

Earlier quoted context omitted.

Agreed. Sprawling system prompts like that are building for the least common denominator, nerfing for anyone or anytime going further.

You do realize that similar biases are also present in the training data?

I do, inevitable, but ime the prompts force certain behaviors at similar strength (instruction following). So it's one thing that the model is biased towards any particular direction by its latent space, it's another that it is biased by an immodifiable prompt which can only be contradicted for the benefit of the lcd at the expense of the more involved operator.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#218

Post by pllbnk »

salad-tycoon wrote: post #216
pllbnk wrote: post #196

Earlier quoted context omitted.

Seems so, unless we manage to pivot to open weight models. Hopefully, Chinese will lead the way along with their consumer hardware. Hard for me to say this because I have always been pro-Western and suddenly it seems like the world has flipped.

I feel the same way for a while now but especially recently. It’s been obvious for a while I suppose but greatly clarified recently. I have just one question for you pllbnk, are we the baddies?

As an European I think Americans and Europeans at large are still on the same page and will be because of the shared cultural ties. Recent economic upheaval (2020-ongoing) just shook the foundations and eroded the trust in the large voter base. Now we are all looking at China and feel a bit envious how stable things look there from afar; they had the Evergrande bankruptcy, media was predicting collapse but they are chugging along; now the big stories are demographics (same as in the West, by the way, so it cancels out) and Taiwan (to me it more and more looks more like Western fearmongering rather than actual danger). Meanwhile they are delivering just what the main voter base in the West needs - affordable goods.

So yeah, at this moment in time it's really really hard to say who are better or worse as the collective West's reputation is tumbling down and China's if not rising, then at least staying put.

Re: Changes in the system prompt between Claude Opus 4.6 and 4.7

#220

Post by mannanj »

gausswho wrote: post #54
mannanj wrote: post #29

Personally, as someone who has been lucky enough to completely cure "incurable" diseases with diet, self experimentation and learning from experts who disagreed with the common societal beliefs at the time - I'm concerned that an AI model and an AI company is planting beliefs and limiting what people can and can't learn through their own will and agency. My concern is these models revert all medical, scientific and p…

While I share your concern for a winners-take-all model getting bent, I do have an optimism that models we've never heard of plug away challenging conclusions in medical canon. We will have a popular vaccine denying AND vaccine authoring models.

Sure. Though which ones will most people use? Do most people use that small obscure vaccine denying or authoring model, is that right to have them use the main societal belief affirming model when it could be wrong?
Post reply on HN