Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's system prompt is over 24k tokens with tools

github.com

Post reply on HN
Print view View original

61–70 of 350 posts

Re: Claude's system prompt is over 24k tokens with tools

#61

Post by otabdeveloper4 »

bjornsing wrote: post #24

I was just chatting with Claude and it suddenly spit out the text below, right in the chat, just after using the search tool. So I'd say the "system prompt" is probably even longer. Claude NEVER repeats, summarizes, or translates song lyrics. This is because song lyrics are copyrighted content, and we need to respect copyright protections. If asked for song lyrics, Claude should decline the request. (There are no son…

Do they actually test these system prompts in a rigorous way? Or is this the modern version of the rain dance?

I don't think you need to spell it out long-form with fancy words like you're a lawyer. The LLM doesn't work that way.

Re: Claude's system prompt is over 24k tokens with tools

#62

Post by abrookewood »

dr_kretyn wrote: post #21

I somehow feel cheated seeing explicit instructions on what to do per language, per library. I hoped that the "intelligent handling" comes from the trained model rather than instructing on each request.

I'm the opposite - I look at how long that prompt is and I'm amazed that the LLM 'understands' it and that it works so well at modifying it's behaviour.

Re: Claude's system prompt is over 24k tokens with tools

#63

Post by [deleted] »

moralestapia wrote: post #59
kergonath wrote: post #55

Earlier quoted context omitted.

> I don't know if anyone has the statistic but I'd guess the immense majority of user queries are like 100 tokens or shorter, imagine loading 24k to solve 0.1k, only a waste of 99.995% of resources. That’s par for the course. These things burn GPU time even when they are used as a glorified version of Google prone to inventing stuff. They are wasteful in the vast majority of cases. > I wish I could just short Anthrop…

>What makes you think the others are significantly different? ChatGPT's prompt is on the order of 1k, if the leaks turn out to be real. Even that one seems a bit high for my taste, but they're the experts, not me. >It’s logical to use the context to tell it what to do. You probably don't know much about this, but no worries I can explain. You can train a model to "become" anything you want, if your default prompt sta…

[deleted]

Re: Claude's system prompt is over 24k tokens with tools

#64

Post by beardedwizard »

moralestapia wrote: post #59
kergonath wrote: post #55

Earlier quoted context omitted.

> I don't know if anyone has the statistic but I'd guess the immense majority of user queries are like 100 tokens or shorter, imagine loading 24k to solve 0.1k, only a waste of 99.995% of resources. That’s par for the course. These things burn GPU time even when they are used as a glorified version of Google prone to inventing stuff. They are wasteful in the vast majority of cases. > I wish I could just short Anthrop…

>What makes you think the others are significantly different? ChatGPT's prompt is on the order of 1k, if the leaks turn out to be real. Even that one seems a bit high for my taste, but they're the experts, not me. >It’s logical to use the context to tell it what to do. You probably don't know much about this, but no worries I can explain. You can train a model to "become" anything you want, if your default prompt sta…

But this is the solution the most cutting edge llm research has yielded, how do you explain that? Are they just willfully ignorant at OpenAI and anthropic? If fine tuning is the answer why aren't the best doing it?

Re: Claude's system prompt is over 24k tokens with tools

#65

Post by mhmmmmmm »

otabdeveloper4 wrote: post #61
bjornsing wrote: post #24

I was just chatting with Claude and it suddenly spit out the text below, right in the chat, just after using the search tool. So I'd say the "system prompt" is probably even longer. Claude NEVER repeats, summarizes, or translates song lyrics. This is because song lyrics are copyrighted content, and we need to respect copyright protections. If asked for song lyrics, Claude should decline the request. (There are no son…

Do they actually test these system prompts in a rigorous way? Or is this the modern version of the rain dance? I don't think you need to spell it out long-form with fancy words like you're a lawyer. The LLM doesn't work that way.

They certainly do, and also offer the tooling to the public: https://docs.anthropic.com/en/docs/build-with-claude/prompt-...

They also recommend to use it to iterate on your own prompts when using Claude Code for example

Re: Claude's system prompt is over 24k tokens with tools

#66

Post by cypherpunks01 »

radeeyate wrote: post #17
eigenblake wrote: post #16

How did they leak it, jailbreak? Was this confirmed? I am checking for the situation where the true instructions are not what is being reported here. The language model could have "hallucinated" its own system prompt instructions, leaving no guarantee that this is the real deal.

All System Prompts from Anthropic models are public information, released by Anthropic themselves: https://docs.anthropic.com/en/release-notes/system-prompts . I'm unsure (I just skimmed through) to what the differences between this and the publicly released ones are, so they're might be some differences.

This system prompt that was posted interestingly includes the result of the US presidential election in November, even though the model's knowledge cutoff date was October. This info wasn't in the anthropic version of the system prompt.

Asking Claude who won without googling, it does seem to know even though it was later than the cutoff date. So the system prompt being posted is supported at least in this aspect.

Re: Claude's system prompt is over 24k tokens with tools

#67

Post by jongjong »

My experience is that as the prompt gets longer, performance decreases. Having such a long prompt with each request cannot be good.

I remember in the early days of OpenAI, they had made the text completion feature available directly and it was much smarter than ChatGPT... I couldn't understand why people were raving about ChatGPT instead of the raw davinci text completion model.

Ir sucks how legal restrictions are dumbing down the models.

Re: Claude's system prompt is over 24k tokens with tools

#68

Post by mcintyre1994 »

dr_kretyn wrote: post #21

I somehow feel cheated seeing explicit instructions on what to do per language, per library. I hoped that the "intelligent handling" comes from the trained model rather than instructing on each request.

I think most of that is about limiting artifacts (code it writes to be previewed in the Claude app) to the supported libraries etc. The trained model can answer questions about and write code in lots of other libraries, but to render correctly in artifacts there’s only a small number of available libraries. And there’ll be all sorts of ways those libraries are imported etc in the training data so it makes sense to tell it how that needs to be done in their environment.

Re: Claude's system prompt is over 24k tokens with tools

#69

Post by aaronbrethorst »

o11c wrote: post #45
behnamoh wrote: post #31

Earlier quoted context omitted.

> The assistant is Claude, created by Anthropic. > The current date is {{currentDateTime}}. > Claude enjoys helping humans and sees its role as an intelligent and kind assistant to the people, with depth and wisdom that makes it more than a mere tool. Why do they refer to Claude in third person? Why not say "You're Claude and you enjoy helping hoomans"?

LLMs are notoriously bad at dealing with pronouns, because it's not correct to blindly copy them like other nouns, and instead they highly depend on the context.

[flagged]

Re: Claude's system prompt is over 24k tokens with tools

#70

Post by ModernMech »

Legend2440 wrote: post #36
4b11b4 wrote: post #10

I like how there are IFs and ELSE IFs but those logical constructs aren't actually explicitly followed... and inside the IF instead of a dash as a bullet point there's an arrow.. that's the _syntax_? hah.. what if there were two lines of instructions, you'd make a new line starting with another arrow..? Did they try some form of it without IFs first?...

Syntax doesn't need to be precise - it's natural language, not formal language. As long as a human could understand it the LLM will too.

Said differently: if it's ambiguous to humans, it will be ambiguous to the LLM too.
Post reply on HN