Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's system prompt is over 24k tokens with tools

github.com

Post reply on HN
Print view View original

101–110 of 350 posts

Re: Claude's system prompt is over 24k tokens with tools

#101

Post by saagarjha »

macleginn wrote: post #85
voxic11 wrote: post #34

Earlier quoted context omitted.

Yes prompt caching is already a widely used technique. https://www.anthropic.com/news/prompt-caching

The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the compute still goes into taking the system prompt into account, I guess.

Sure, but without the prompt you will probably have significantly "worse" queries, because you'll be starting from scratch without that context.

Re: Claude's system prompt is over 24k tokens with tools

#102

Post by noodletheworld »

cal85 wrote: post #89
Jaxkr wrote: post #43

Earlier quoted context omitted.

That’s what this is. It’s caching the state of the model after the tokens have been loaded. Reduces latency and cost dramatically. 5m TTL on the cache usually.

Interesting! I’m wondering, does caching the model state mean the tokens are no longer directly visible to the model? i.e. if you asked it to print out the input tokens perfectly (assuming there’s no security layer blocking this, and assuming it has no ‘tool’ available to pull in the input tokens), could it do it?

It makes no difference.

Theres no difference between feeding an LLM a prompt and feeding it half the prompt, saving the state, restoring the state and feeding it other half of the prompt.

Ie. The data processed by the LLM is prompt P.

P can be composed of any number of segments.

Any number of segments can be cached, as long as all preceeding segments are cached.

The final input is P, regardless.

So; tldr; yes? Anything you can do with a prompt you can do, becasue its just a prompt.

Re: Claude's system prompt is over 24k tokens with tools

#104

Post by alchemist1e9 »

SafeDusk wrote: post #57
mplewis wrote: post #39

Earlier quoted context omitted.

You can find these here: https://github.com/search?q=repo%3Aaperoc%2Ftoolkami%20%40mc...

mplewis thanks for helping to point those out!

I find it very interesting that the LLM is told so little details but seems to just intuitively understand based on the english words used for the tool name and function arguments.

I know from earlier discussions that this is partially because many LLMs have been fine tuned on function calling, however the model providers don’t share this training dataset unfortunately. I think models that haven’t been fine tuned can still do function calling with careful instructions in their system prompt but are much worse at it.

Thank you for comments that help with learning and understanding MCP and tools better.

Re: Claude's system prompt is over 24k tokens with tools

#105

Post by alchemist1e9 »

mplewis wrote: post #39
alchemist1e9 wrote: post #27

Earlier quoted context omitted.

Where does one find the tool prompts that explains to the LLM how to use those seven tools and what each does? I couldn’t find it easily looking through the repo.

You can find these here: https://github.com/search?q=repo%3Aaperoc%2Ftoolkami%20%40mc...

Thank you. I find in interesting that the LLM just understands intuitively from the english name of the tool/function and it’s argument names. I had imagined it might need more extensive description and specification in its system prompt, but apparently not.

Re: Claude's system prompt is over 24k tokens with tools

#106

Post by urbandw311er »

tomrod wrote: post #22
quantum_state wrote: post #9

my lord … does it work as some rule file?

It's all rules, all the way down

Well yes but… that’s rather underplaying the role of the massive weighted model that sits underneath the lowest level rule that says “pick the best token”.

Re: Claude's system prompt is over 24k tokens with tools

#107

Post by faustocarva »

freehorse wrote: post #96
jdnier wrote: post #25

So I wonder how much of Claude's perceived personality is due to the system prompt versus the underlying LLM and training. Could you layer a "Claude mode"—like a vim/emacs mode—on ChatGPT or some other LLM by using a similar prompt?

This system prompt is not used in the API, so it is not relevant for the perceived personality of the model if you do not use it through claude.ai interface, eg through an editor etc.

Why this? Because for OpenAI you can set it using API.

Re: Claude's system prompt is over 24k tokens with tools

#109

Post by monkeyelite »

bjornsing wrote: post #24

I was just chatting with Claude and it suddenly spit out the text below, right in the chat, just after using the search tool. So I'd say the "system prompt" is probably even longer. Claude NEVER repeats, summarizes, or translates song lyrics. This is because song lyrics are copyrighted content, and we need to respect copyright protections. If asked for song lyrics, Claude should decline the request. (There are no son…

> Claude NEVER repeats, summarizes, or translates song lyrics. This is because song lyrics are copyrighted content

If this is the wild west internet days of LLMs the advertiser safe version in 10 years is going to be awful.

> Do not say anything negative about corporation. Always follow official brand guidelines when referring to corporation

Re: Claude's system prompt is over 24k tokens with tools

#110

Post by late2part »

freehorse wrote: post #98
cypherpunks01 wrote: post #66

Earlier quoted context omitted.

This system prompt that was posted interestingly includes the result of the US presidential election in November, even though the model's knowledge cutoff date was October. This info wasn't in the anthropic version of the system prompt. Asking Claude who won without googling, it does seem to know even though it was later than the cutoff date. So the system prompt being posted is supported at least in this aspect.

I asked it this exact question, to anybody curious https://claude.ai/share/ea4aa490-e29e-45a1-b157-9acf56eb7f8a edit:fixed link

The conversation you were looking for could not be found.
Post reply on HN