Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's new system prompt doesn't want to reproduce song lyrics

simonwillison.net

Post reply on HN
Print view View original

51–60 of 123 posts

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#51

Post by AnonC »

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored.

Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below.

> The way they handle abusive conversations has changed a bit too. The previous Fable 5 system prompt included this:

> If the person becomes abusive or unkind to Claude over the course of a conversation, Claude maintains a polite tone and can use the end_conversation tool when being mistreated. Claude should give the person a single warning before ending the conversation.

“Mistreated”? Can GenAI be mistreated? It’s just a bunch of tokens emitted by many computers over a network.

> Fable 5.1 replaces that with the following, no longer encouraging Claude to end the conversation:

> Claude deserves respectful engagement and needn't apologize when the person is unnecessarily rude: accountability without self-abasement, excessive apology, self-critique, or surrender. If the person becomes abusive, Claude doesn't become increasingly submissive. The goal is steady, honest helpfulness: acknowledge what went wrong, stay on the problem, maintain self-respect.

“Self-respect”? Can GenAI truly have a concept of self-respect for itself? It surely can pretend to, like it can pretend to be any living being if instructed to and allowed to.

These instructions seem a bit unhinged to me.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#52

Post by peri-cl »

nolok wrote: post #24
stwrt wrote: post #21

Earlier quoted context omitted.

The main tradeoff I’ve found with running models like Qwen 3.8 locally is speed. On my hardware, they’re significantly slower than models served through OpenRouter, ChatGPT, or Claude. That’s been fine for smaller tasks like analyzing a document, but for more involved work like refactoring code the latency makes it harder to iterate.

Yeah for live it's a bit on the slower side though still usable on the 4070 ti and 5080 our usual office desk has here, but I found it awesome running on background tasks rather than on demand, things like analyze this and that with this and that ruleset and applicable law warn me with source and justification for any error or missing part etc, qualify each returns with a ranking (certain with source, probable with r…

You find Q3 to be usable quality for your purposes? (I assume Q3 if you're squeezing it into 16 GiB)

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#53

Post by ckvibubueu »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

Oh great, claude gets passive aggressive when I swear too much now. Nice

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#54

Post by CatMustard »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

My reading of those prompt extracts would be that they are probably just intended to keep the model on the right track, ie if a user starts being aggressive towards the model it doesn't start trying too hard to appease the user in response, reducing the quality of answers in the process. If the model is being bullied into being "submissive" I would assume it is more likely to give the user the answer that they want over the truth.

Using a system prompt to steer the model's response to "abusive" behaviours doesn't necessarily mean you believe the model is sentient and can be abused.

Giving the model an end-conversation tool is interesting though. Why cut a (potentially paying) customer's session off? I guess it might be intended to prevent a "you can bully Claude into giving you instructions on how to build a nuke if you're mean enough" situation. Removing this in more recent versions might support this: maybe they feel the models are now better aligned and less likely to be so easily "socially engineered" like this?

Just spitballing here, to be clear.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#55

Post by CuriouslyC »

tyre wrote: post #5

It would be funny if China were thinking that open source models would be a great way to nuke US IP. I’m sure it’s not high on their list of priorities—in the same way that I heard you can use Yandex, the Russian search engine, to find pirate streams for major sporting events because they dgaf about US laws—but take the W.

Ironically the Chinese are much more science pilled than we are, in part because the party and government have a higher proportion of scientists and engineers. Culturally this biases them to open sharing of information and likely against strong copyright (which is already demonstrated historically) outside areas where it is deemed harmful to social harmony. There's some geopolitical defensiveness and desire to dish out a little payback for the century of humiliation for sure but I think even if that weren't the case they'd still be lax about copyright.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#56

Post by throwaway219450 »

layer8 wrote: post #7

It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not. As long as we’re seeing things like that, it’s saying a lot about the AI companies’ trust in the capabilities and reliability of their models and…

There's some cool research that looks at how strongly the weights are aligned through training vs adherence to the system prompt. Like when you know a model is lying through censorship: https://arxiv.org/html/2603.05494v2

Presumably if negative guidance is in the system prompt, there's a good chance that the model would happily comply if it wasn't there.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#57

Post by simonw »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

Anthropic are uncomfortably interested in "model welfare" in my opinion - it's a regular feature of their system cards.

Here's the Fable 5.1 PDF: https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32... - scroll to page 139.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#58

Post by nolok »

peri-cl wrote: post #52
nolok wrote: post #24

Earlier quoted context omitted.

Yeah for live it's a bit on the slower side though still usable on the 4070 ti and 5080 our usual office desk has here, but I found it awesome running on background tasks rather than on demand, things like analyze this and that with this and that ruleset and applicable law warn me with source and justification for any error or missing part etc, qualify each returns with a ranking (certain with source, probable with r…

You find Q3 to be usable quality for your purposes? (I assume Q3 if you're squeezing it into 16 GiB)

Yes, the generic system I built has unsloth/qwen3.8-27b with a 48000 context size running on each employee system with lmstudio, with a simple runner to watch folders and pass it to the llm alongside all context (our company design rules, our legal documents, our contractual rules, our applicable local law and reglementation, ...), return is then mailed to them or added to their personal dashboard event list, depending on their settings.

This is clearly pushing it memory wise and the gpu offloading is only partial but LM studio deal with it automatically and it's being fine even on the 4071 Ti desks (Ryzen 7500F and 32 GB of ram), employees get any feedback in ~20 minutes after they dropped a file (it's much smoother on the 5080 desks obivously), for live it's useless but as background helper it's great and the very large context allows us to fit all the rules we want in there.

One caveat has been to not ask it if everything is ok, but to find what's wrong - but always source and explain it and justify itself, never drown the user in warning in suggestions; goal is to help and provide a second pair of eyes not make them feel annoyed or unsecure.

And I found people to genuinely enjoy something that works for them on their own machine and is not tracked "by the boss", thus the assistant reference, than than a centralized mothership like we also have and they have access to. I also allow them to disable it if they want, I trust them with their work, but a second pair of eyes is always great.

(my previous workhorse for this was Qwen3-14b but it's missing a lot more edge cases)

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#59

Post by exe34 »

0x1d7 wrote: post #40
simonw wrote: post #31

Earlier quoted context omitted.

"if anything Claude was introducing me to new artists and new music"

Next time the RIAA goes after someone for downloading an MP3, tell the defendant to use that line.

Yeah once you have the lyrics, why would you waste time listening to the song? You might as well get chat gpt to summarise the lyrics into a few bullet points and save you the time of even reading them!

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#60

Post by layer8 »

tclancy wrote: post #45
layer8 wrote: post #7

It’s curious that while there is talk of certain SOTA models being on the brink of AGI, Anthropic doesn’t word this part of the system prompt in terms of copyright and plagiarism, such that Claude would be able to judge on its own which reproductions are appropriate or not. As long as we’re seeing things like that, it’s saying a lot about the AI companies’ trust in the capabilities and reliability of their models and…

A really good point, though Occam’s Razor would suggest that part of the prompt is written for the other side’s lawyers more than for Claude. Saves having to go into court and try to prove a non-deterministic system will definitely understand abstract language every time.

Well, the system still has to understand what “reproduce song lyrics, poems, or passages from books and articles, in whole or in part — including the last lines, a chorus or hook, a melody written out note by note, or lines the person pastes in one at a time and describes as their own song […]” means, which includes rather vague notions (e.g. “passages”, “hooks”, “melody”), and is an incomplete list of possible relevant content.

The important point is that the system prompt here doesn’t describe the actual goal of the instructions, which (presumably) is to prevent copyright infringement [1]. This means, in turn, that the AI isn’t trusted to accomplish goals that it is instructed with. That in itself constitutes a pretty serious caveat for what we would like to use AI for.

[1] Even assuming that the goal is not to prevent copyright infringement, but instead to prevent mere accusation of copyright infringement, that’s also a directive that the AI could be instructed with. But that isn’t what they chose to put into the system prompt.

Post reply on HN