Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's system prompt is over 24k tokens with tools

github.com

Post reply on HN
Print view View original

51–60 of 350 posts

Re: Claude's system prompt is over 24k tokens with tools

#51

Post by katzenversteher »

rdtsc wrote: post #47
behnamoh wrote: post #31

Earlier quoted context omitted.

> The assistant is Claude, created by Anthropic. > The current date is {{currentDateTime}}. > Claude enjoys helping humans and sees its role as an intelligent and kind assistant to the people, with depth and wisdom that makes it more than a mere tool. Why do they refer to Claude in third person? Why not say "You're Claude and you enjoy helping hoomans"?

> Why do they refer to Claude in third person? Why not say "You're Claude and you enjoy helping hoomans"? But why would they say that? To me that seems a bit childish. Like, say, when writing a script do people say "You're the program, take this var. You give me the matrix"? That would look goofy.

"It puts the lotion on the skin, or it gets the hose again"

Re: Claude's system prompt is over 24k tokens with tools

#52

Post by katzenversteher »

horacemorace wrote: post #44
behnamoh wrote: post #31

Earlier quoted context omitted.

> The assistant is Claude, created by Anthropic. > The current date is {{currentDateTime}}. > Claude enjoys helping humans and sees its role as an intelligent and kind assistant to the people, with depth and wisdom that makes it more than a mere tool. Why do they refer to Claude in third person? Why not say "You're Claude and you enjoy helping hoomans"?

LLMs don’t seem to have much notion of themselves as a first person subject, in my limited experience of trying to engage it.

From their perspective they don't really know who put the tokens there. They just caculated the probabilities and then the inference engine adds tokens to the context window. Same with user and system prompt, they just appear in the context window and the LLM just gets "user said: 'hello', assistant said: 'how can I help '" and it just calculates the probabilities of the next token. If the context window had stopped in the user role it would have played the user role (calculated the probabilities for the next token of the user).

Re: Claude's system prompt is over 24k tokens with tools

#53

Post by potholereseller »

dr_kretyn wrote: post #21

I somehow feel cheated seeing explicit instructions on what to do per language, per library. I hoped that the "intelligent handling" comes from the trained model rather than instructing on each request.

When you've trained your model on all available data, the only things left to improve are the training algorithm and the system prompt; the latter is far easier and faster to tweak. The system prompts may grow yet more, but they can't exceed the token limit. To exceed that limit, they may create topic-specific system prompts, selected by another, smaller system prompt, using the LLM twice:

user's-prompt + topic-picker-prompt -> LLM -> topic-specific-prompt -> LLM

This will enable the cumulative size of system prompts to exceed the LLM's token limit. But this will only occur if we happen to live in a net-funny universe, which physicists have not yet determined.

Re: Claude's system prompt is over 24k tokens with tools

#54

Post by Oras »

jdnier wrote: post #25

So I wonder how much of Claude's perceived personality is due to the system prompt versus the underlying LLM and training. Could you layer a "Claude mode"—like a vim/emacs mode—on ChatGPT or some other LLM by using a similar prompt?

Training data matters. They used lots of xml like tags to structure the training data. You can see that in the system prompt.

Re: Claude's system prompt is over 24k tokens with tools

#55

Post by kergonath »

moralestapia wrote: post #46

[flagged]

> I don't know if anyone has the statistic but I'd guess the immense majority of user queries are like 100 tokens or shorter, imagine loading 24k to solve 0.1k, only a waste of 99.995% of resources.

That’s par for the course. These things burn GPU time even when they are used as a glorified version of Google prone to inventing stuff. They are wasteful in the vast majority of cases.

> I wish I could just short Anthropic.

What makes you think the others are significantly different? If all they have is a LLM screwdriver, they’re going to spend a lot of effort turning every problem into a screw, it’s not surprising. A LLM cannot reason, just generate text depending on the context. It’s logical to use the context to tell it what to do.

Re: Claude's system prompt is over 24k tokens with tools

#56

Post by SafeDusk »

swyx wrote: post #42
SafeDusk wrote: post #14

In addition to having long system prompts, you also need to provide agents with the right composable tools to make it work. I’m having reasonable success with these seven tools: read, write, diff, browse, command, ask, think. There is a minimal template here if anyone finds it useful: https://github.com/aperoc/toolkami

> 18 hours ago you just released this ? lol good timing

I did! Thanks for responding and continue to do your great work, I'm a fan as a fellow Singaporean!

Re: Claude's system prompt is over 24k tokens with tools

#57

Post by SafeDusk »

mplewis wrote: post #39
alchemist1e9 wrote: post #27

Earlier quoted context omitted.

Where does one find the tool prompts that explains to the LLM how to use those seven tools and what each does? I couldn’t find it easily looking through the repo.

You can find these here: https://github.com/search?q=repo%3Aaperoc%2Ftoolkami%20%40mc...

mplewis thanks for helping to point those out!

Re: Claude's system prompt is over 24k tokens with tools

#58

Post by SafeDusk »

triyambakam wrote: post #28
SafeDusk wrote: post #14

In addition to having long system prompts, you also need to provide agents with the right composable tools to make it work. I’m having reasonable success with these seven tools: read, write, diff, browse, command, ask, think. There is a minimal template here if anyone finds it useful: https://github.com/aperoc/toolkami

Really interesting, thank you

Hope you find it useful, feel free to reach out if you need help or think it can be made better.

Re: Claude's system prompt is over 24k tokens with tools

#59

Post by moralestapia »

kergonath wrote: post #55
moralestapia wrote: post #46

[flagged]

> I don't know if anyone has the statistic but I'd guess the immense majority of user queries are like 100 tokens or shorter, imagine loading 24k to solve 0.1k, only a waste of 99.995% of resources. That’s par for the course. These things burn GPU time even when they are used as a glorified version of Google prone to inventing stuff. They are wasteful in the vast majority of cases. > I wish I could just short Anthrop…

>What makes you think the others are significantly different?

ChatGPT's prompt is on the order of 1k, if the leaks turn out to be real. Even that one seems a bit high for my taste, but they're the experts, not me.

>It’s logical to use the context to tell it what to do.

You probably don't know much about this, but no worries I can explain. You can train a model to "become" anything you want, if your default prompt starts to be measured in kilobytes, it might as well be better to re-train (obv. not re-train the same one, but v2.1 or whatever, train it with this in mind) and/or fine tune, because your model behaves quite different from what you want it to do.

I don't know the exact threshold, there might not even be one as training and LLM takes some sort of artisan skills, but if you need 24k just to boot the thing you're clearly doing something wrong, aside from the waste of resources.

Re: Claude's system prompt is over 24k tokens with tools

#60

Post by azinman2 »

FooBarWidget wrote: post #49
eigenblake wrote: post #16

How did they leak it, jailbreak? Was this confirmed? I am checking for the situation where the true instructions are not what is being reported here. The language model could have "hallucinated" its own system prompt instructions, leaving no guarantee that this is the real deal.

The other day I was talking to Grok, and then suddenly it started outputting corrupt tokens, after which it outputted the entire system prompt. I didn't ask for it. There truly are a million ways for LLMs to leak their system prompt.

What did it say?
Post reply on HN