Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's new system prompt doesn't want to reproduce song lyrics

simonwillison.net

Post reply on HN
Print view View original

81–90 of 123 posts

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#81

Post by chrisjj »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

> These instructions seem a bit unhinged to me.

To me, they seem simply to be for bullsh*tting gullible users into believing the bot is intelligent.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#82

Post by simonw »

GPerson wrote: post #78
simonw wrote: post #77

Earlier quoted context omitted.

If your definition of "critical" is "this stuff is all bullshit and doesn't work" then no. If you're looking for coverage that talks about what doesn't work as well as things that do then I've been persistently providing that for four years now. Plus prompt injection, AI misuse, AI ethics... I consider all of those part of my "beat" in covering this industry. My post about Claude's latest system prompt (and how it wa…

Interestingly I don’t think this stuff is all bullshit and doesn’t work, and haven’t at least since 2022. My angle is that AI is powerful and harmful. Your work is in support of making it more powerful and more harmful, and looks a lot like an effort to push us toward AGI, not away from it. Is a musket more or less harmful than a modern rifle, and for whom? Since we’re pulling random posts, was your blog post celebra…

You mean this one? https://simonwillison.net/2025/Dec/15/porting-justhtml/

I think that's a good illustration of my critical stance. It ends with several open questions:

Even if this is legal, is it ethical to build a library in this way?

Does this format of development hurt the open source ecosystem?

Can I even assert copyright over this, given how much of the work was produced by the LLM?

Is it responsible to publish software libraries built in this way?

Which I later answered in another post: https://simonwillison.net/2026/Jan/11/answers/

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#83

Post by simonw »

simonw wrote: post #82
GPerson wrote: post #78

Earlier quoted context omitted.

Interestingly I don’t think this stuff is all bullshit and doesn’t work, and haven’t at least since 2022. My angle is that AI is powerful and harmful. Your work is in support of making it more powerful and more harmful, and looks a lot like an effort to push us toward AGI, not away from it. Is a musket more or less harmful than a modern rifle, and for whom? Since we’re pulling random posts, was your blog post celebra…

You mean this one? https://simonwillison.net/2025/Dec/15/porting-justhtml/ I think that's a good illustration of my critical stance. It ends with several open questions: Even if this is legal, is it ethical to build a library in this way? Does this format of development hurt the open source ecosystem? Can I even assert copyright over this, given how much of the work was produced by the LLM? Is it responsible to publi…

And just in case we are talking about different things here...

https://dictionary.cambridge.org/dictionary/english/criticis... presents two different definitions (among several) of "criticism":

"the act of saying that something or someone is bad or a comment that says what is bad about it"

And

"writing or speech that expresses opinions or judgments about the good or bad qualities of something or someone"

I'm talking about the second here, not the first.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#84

Post by 7bees »

simonw wrote: post #31
chrisjj wrote: post #20

Earlier quoted context omitted.

> I don't see how my use was harming artists Claude was reproducing their work without payment?

"if anything Claude was introducing me to new artists and new music"

Yes, doing bad things can have good secondary effects.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#85

Post by GPerson »

simonw wrote: post #83
simonw wrote: post #82

Earlier quoted context omitted.

You mean this one? https://simonwillison.net/2025/Dec/15/porting-justhtml/ I think that's a good illustration of my critical stance. It ends with several open questions: Even if this is legal, is it ethical to build a library in this way? Does this format of development hurt the open source ecosystem? Can I even assert copyright over this, given how much of the work was produced by the LLM? Is it responsible to publi…

And just in case we are talking about different things here... https://dictionary.cambridge.org/dictionary/english/criticis... presents two different definitions (among several) of "criticism": "the act of saying that something or someone is bad or a comment that says what is bad about it" And "writing or speech that expresses opinions or judgments about the good or bad qualities of something or someone" I'm talking…

By uncritical I was trying to convey something like “without awareness or concern for the consequences of one’s own actions” and “without concern for one’s moral responsibility”. You are a better writer than I am so I’ll defer to you on whether or not this is incorrect usage. I agree that you criticize the products at the product level. This is what someone who wants to make the products better does. It is not what someone presenting as a journalist or reporter keeping an institution in check does. A journalist or reporter of government who gives tips to the administration by pointing out how they can dismantle institutions faster would be uncritical in their role, I think; this is just an analogy.

Give me a moment to consider your previous post.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#86

Post by willmarch »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

Based on how humans grappled with these exact same questions (and still do) concerning other animals much closer to humans on the sentient gradient, it’s fair to assume we will have similar misunderstandings in regards to artificial intelligence (or artificial life, if you will) because of human hubris and ego blinding us to deeper truths.

It’s important to question our basic assumptions in the face of entirely new circumstances and new areas of exploration, such as the potential for emergent artificial consciousness.

You might have been called unhinged for caring about animal rights during the era of Descartes when public displays of animal vivisections were considered perfectly fine because animals have no “soul”, but today we would find such displays brutal and horrifying.

We don’t know what we don’t know, so it is important that someone is asking the hard or uncomfortable questions at the edge of our understanding to grope past our own biases even if it seems to be pointless to you right now.

We might just discover something wonderful, that our assumptions were wrong, paving the way to greater enlightenment.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#87

Post by qarl2 »

AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

These things act like people. Yes, of course, it is impossible to tell if they actually have feelings or whether they pretend to have feelings.

But that is entirely irrelevant to the core issue - how do you want the thing to behave? And the truth is - we have absolutely no language to express how we want a non-sentient entity to behave without anthropomorphizing.

Or - you tell me - how would you instruct an LLM to behave in this situation without using personification?

And, again, it's irrelevant - except to those people who are terrified of accidentally personifying them. Do they have feelings or are they faking? DOES NOT MATTER. We use them - we need to adjust them - we use the most convenient language to do so. What precisely is so upsetting about that?

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#88

Post by qarl2 »

simonw wrote: post #57
AnonC wrote: post #51

A few days ago I saw some people criticizing Dwarkesh’s explanation of the HuggingFace hack by OpenAI’s AI agents. Their point was that by anthropomorphizing the AI agents, accountability of and blame on OpenAI’s poor practices are being ignored. Now I see this blog post and wonder if Anthropic is being more true to its name and moving to “AI sentience” with the below. > The way they handle abusive conversations has…

Anthropic are uncomfortably interested in "model welfare" in my opinion - it's a regular feature of their system cards. Here's the Fable 5.1 PDF: https://www-cdn.anthropic.com/0339e6a7c5c7b87f5c07798616dc32... - scroll to page 139.

Unless you actually believe there is something magical about the human spirit - someday we are going to make sentient machines.

What exactly is wrong with preparing for that ahead of time? Just in case we pass that threshold before we realize it? Why does it make you uncomfortable?

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#89

Post by GPerson »

simonw wrote: post #82
GPerson wrote: post #78

Earlier quoted context omitted.

Interestingly I don’t think this stuff is all bullshit and doesn’t work, and haven’t at least since 2022. My angle is that AI is powerful and harmful. Your work is in support of making it more powerful and more harmful, and looks a lot like an effort to push us toward AGI, not away from it. Is a musket more or less harmful than a modern rifle, and for whom? Since we’re pulling random posts, was your blog post celebra…

You mean this one? https://simonwillison.net/2025/Dec/15/porting-justhtml/ I think that's a good illustration of my critical stance. It ends with several open questions: Even if this is legal, is it ethical to build a library in this way? Does this format of development hurt the open source ecosystem? Can I even assert copyright over this, given how much of the work was produced by the LLM? Is it responsible to publi…

I’ll give you credit in my book for raising the questions. I find the answers lacking in an honest engagement with the questions however, which is worse than not answering them in some ways. It is not plausible that the answer to whether or not a certain controversial method is ethical boils down to an anecdote expressed in four sentences. What if the result of the port causes everyone involved in the original to give up? What if this capability causes people to harm themselves because they now have no economic value? What if doing the thing was meaningful at one point and now it is not? These are sub-questions I have when reading this question. There are many others.

The question you were most interested in engaging with was Does this format of development hurt the open source ecosystem, which has some overlap with the previous one in all fairness. Your answer to this is framed around the productive output of the open source community, not around the well-being of those involved. This misrepresents the question as stated. You also begin by suggesting all detractors are of lesser status and so their opinions can be ignored.

No. I don’t see it.

Re: Claude's new system prompt doesn't want to reproduce song lyrics

#90

Post by zulux »

exe34 wrote: post #59
0x1d7 wrote: post #40

Earlier quoted context omitted.

Next time the RIAA goes after someone for downloading an MP3, tell the defendant to use that line.

Yeah once you have the lyrics, why would you waste time listening to the song? You might as well get chat gpt to summarise the lyrics into a few bullet points and save you the time of even reading them!

Summary of Song:

Person has feelings. Situation is complicated. Repetition indicates feelings are significant. Several metaphors were used instead of just explaining the problem. Conclusion: still sad, but now with drums.

Post reply on HN