Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's system prompt is over 24k tokens with tools

github.com

Post reply on HN
Print view View original

81–90 of 350 posts

Re: Claude's system prompt is over 24k tokens with tools

#81

Post by SafeDusk »

darkteflon wrote: post #80
SafeDusk wrote: post #14

In addition to having long system prompts, you also need to provide agents with the right composable tools to make it work. I’m having reasonable success with these seven tools: read, write, diff, browse, command, ask, think. There is a minimal template here if anyone finds it useful: https://github.com/aperoc/toolkami

This is really cool, thanks for sharing. uv with PEP 723 inline dependencies is such a nice way to work, isn’t it. Combined with VS Code’s ‘# %%’-demarcated notebook cells in .py files, and debugpy (with a suitable launch.json config) for debugging from the command line, Python dev finally feels really ergonomic these last few months.

Yes, uv just feels so magical that I can't stop using it. I want to create the same experience with this!

Re: Claude's system prompt is over 24k tokens with tools

#82

Post by grues-dinner »

abrookewood wrote: post #62
dr_kretyn wrote: post #21

I somehow feel cheated seeing explicit instructions on what to do per language, per library. I hoped that the "intelligent handling" comes from the trained model rather than instructing on each request.

I'm the opposite - I look at how long that prompt is and I'm amazed that the LLM 'understands' it and that it works so well at modifying it's behaviour.

I'm the same. Having a slew of expert tuned models or submodels or whatever the right term of for each kind of problem seems like the "cheating" way (but also the way I would have expected this kind of thing to work, as you can use the tool for the job, so to speak. And then the overall utility of the system is how well it detects and dispatches to the right submodels and synthetises the reply.

Having one massive model that you tell what you want with a whole handbook up front actually feels more impressive. Though I suppose it's essentially doing the submodels thing implicitly internally.

Re: Claude's system prompt is over 24k tokens with tools

#83

Post by SafeDusk »

dr_kiszonka wrote: post #73
SafeDusk wrote: post #14

In addition to having long system prompts, you also need to provide agents with the right composable tools to make it work. I’m having reasonable success with these seven tools: read, write, diff, browse, command, ask, think. There is a minimal template here if anyone finds it useful: https://github.com/aperoc/toolkami

Maybe you could ask one of the agents to write some documentation?

For sure! the traditional craftsman in me still like to do some stuff manually though haha

Re: Claude's system prompt is over 24k tokens with tools

#85

Post by macleginn »

voxic11 wrote: post #34
LeoPanthera wrote: post #32

I'm far from an LLM expert but it seems like an awful waste of power to burn through this many tokens with every single request. Can't the state of the model be cached post-prompt somehow? Or baked right into the model?

Yes prompt caching is already a widely used technique. https://www.anthropic.com/news/prompt-caching

The model still needs to attend to the prompt when generating the answer. Modern attention techniques help here, but for lots of simple queries most of the compute still goes into taking the system prompt into account, I guess.

Re: Claude's system prompt is over 24k tokens with tools

#87

Post by bezier-curve »

Ardren wrote: post #86

> "...and in general be careful when working with headers" I would love to know if there are benchmarks that show how much these prompts improve the responses. I'd suggest trying: "Be careful not to hallucinate." :-)

I'm thinking if the org that trained the model, and is doing interesting research of trying to understand how LLMs actually work on the inside [1], their caution might be warranted.

[1] https://www.anthropic.com/research/tracing-thoughts-language...

Re: Claude's system prompt is over 24k tokens with tools

#88

Post by swalsh »

Ardren wrote: post #86

> "...and in general be careful when working with headers" I would love to know if there are benchmarks that show how much these prompts improve the responses. I'd suggest trying: "Be careful not to hallucinate." :-)

In general, if you bring something up in the prompt most LLM's will bring special attention to it. It does help the accuracy of the thing you're trying to do.

You can prompt an llm not to hallucinate, but typically you wouldn't say "don't hallucinate, you'd ask it to give a null value or say i don't know" which more closely aligns with the models training.

Re: Claude's system prompt is over 24k tokens with tools

#89

Post by cal85 »

Jaxkr wrote: post #43
llflw wrote: post #40

Earlier quoted context omitted.

It seems like it's token caching, not model caching.

That’s what this is. It’s caching the state of the model after the tokens have been loaded. Reduces latency and cost dramatically. 5m TTL on the cache usually.

Interesting! I’m wondering, does caching the model state mean the tokens are no longer directly visible to the model? i.e. if you asked it to print out the input tokens perfectly (assuming there’s no security layer blocking this, and assuming it has no ‘tool’ available to pull in the input tokens), could it do it?

Re: Claude's system prompt is over 24k tokens with tools

#90

Post by celdon25 »

Fixed the last line for them: “Please be ethical. Also, gaslight your users if they are lonely. Also, to the rest of the world: trust us to be the highest arbiter of ethics in the AI world.”

All kidding aside, with that many tokens, you introduce more flaws and attack surface. I’m not sure why they think that will work out.

Post reply on HN