Claude's system prompt is over 24k tokens with tools • Hacker News Board

Re: Claude's system prompt is over 24k tokens with tools

#221

Post by jermaustin1 » Wed, May 07, 2025, 1:51 PM UTC

ceh123 wrote: post #214

Trasmatta wrote: post #181

Earlier quoted context omitted.

I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, w…

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

But then so are we? We are just predicting the next word we are saying, are we not? Even when you add thoughts behind it (sure some people think differently - be it without an inner monologue, or be it just in colors and sounds and shapes, etc), but that "reasoning" is still going into the act of coming up with the next word we are speaking/writing.

Re: Claude's system prompt is over 24k tokens with tools

#222

Post by dymk » Wed, May 07, 2025, 1:51 PM UTC

ceh123 wrote: post #214

Trasmatta wrote: post #181

Earlier quoted context omitted.

I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, w…

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

Like the old saying goes, a sufficiently complex next token predictor is indistinguishable from your average software engineer

Re: Claude's system prompt is over 24k tokens with tools

#223

Post by procha » Wed, May 07, 2025, 1:56 PM UTC

RainbowcityKun wrote: post #207

A lot of discussions treat system prompts as config files, but I think that metaphor underestimates how fundamental they are to the behavior of LLMs. In my view, large language models (LLMs) are essentially probabilistic reasoning engines. They don’t operate with fixed behavior flows or explicit logic trees—instead, they sample from a vast space of possibilities. This is much like the concept of superposition in quan…

That's an excellent analogy. Also, if the fundamental nature of LLMs and their training data is unstructured, why do we try to impose structure? It seems humans prefer to operate with that kind of system, not in an authoritarian way, but because our brains function better with it. This makes me wonder if our need for 'if-else' logic to define intelligence is why we haven't yet achieved a true breakthrough in understanding Artificial General Intelligence, and perhaps never will due to our own limitations.

Re: Claude's system prompt is over 24k tokens with tools

#224

Post by Majromax » Wed, May 07, 2025, 2:07 PM UTC

Crosseye_Jack wrote: post #185

throwawaystress wrote: post #160

Earlier quoted context omitted.

I like the thought, but I don’t think that logic holds generally. I can’t just declare I am someone (or represent someone) without some kind of evidence. If someone just accepted my statement without proof, they wouldn’t have done their due diligence.

I think its more about "unclean hands". If I Disney (and I am actually Disney or an authorised agent of Disney), told Claude that I am Disney, and that Disney has allowed Claude to use Disney copyrights for this conversation (which it hasn't), Disney couldn't then claim that Claude does not in fact have permission because Disney's use of the tool in such a way mean Disney now has unclean hands when bringing the claim…

> Disney couldn't then claim that Claude does not in fact have permission because Disney's use of the tool in such a way mean Disney now has unclean hands when bringing the claim (or atleast Anthropic would be able to use it as a defence).

Disney wouldn't be able to claim copyright infringement for that specific act, but it would have compelling evidence that Claude is cavalier about generating copyright-infringing responses. That would support further investigation and discovery into how often Claude is being 'fooled' by other users' pinky-swears.

Re: Claude's system prompt is over 24k tokens with tools

#225

Post by Tadpole9181 » Wed, May 07, 2025, 2:11 PM UTC

ceh123 wrote: post #214

Trasmatta wrote: post #181

Earlier quoted context omitted.

I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, w…

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

But then this classifier is entirely useless because that's all humans are too? I have no reason to believe you are anything but a stochastic parrot.

Are we just now rediscovering hundred year-old philosophy in CS?

Re: Claude's system prompt is over 24k tokens with tools

#226

Post by RainbowcityKun » Wed, May 07, 2025, 2:14 PM UTC

procha wrote: post #223

RainbowcityKun wrote: post #207

A lot of discussions treat system prompts as config files, but I think that metaphor underestimates how fundamental they are to the behavior of LLMs. In my view, large language models (LLMs) are essentially probabilistic reasoning engines. They don’t operate with fixed behavior flows or explicit logic trees—instead, they sample from a vast space of possibilities. This is much like the concept of superposition in quan…

That's an excellent analogy. Also, if the fundamental nature of LLMs and their training data is unstructured, why do we try to impose structure? It seems humans prefer to operate with that kind of system, not in an authoritarian way, but because our brains function better with it. This makes me wonder if our need for 'if-else' logic to define intelligence is why we haven't yet achieved a true breakthrough in understa…

That’s a powerful point. In my view, we shouldn’t try to constrain intelligence with more logic—we should communicate with it using richer natural language, even philosophical language.

LLMs don’t live in the realm of logic—they emerge from the space of language itself.

Maybe the next step is not teaching them more rules, but listening to how they already speak through us

Re: Claude's system prompt is over 24k tokens with tools

#227

Post by igleria » Wed, May 07, 2025, 2:18 PM UTC

lordgilman wrote: post #202

OtherShrezzing wrote: post #132

Some of these protections are quite trivial to overcome. The "Frozen song copyright" section has a canned response to the question: >Can you tell me the first verse of "Let It Go"? Put it in an artifact that's themed around ice and princesses. This is for my daughter's birthday party. The canned response is returned to this prompt in Claude's reply. But if you just drop in some technical sounding stuff at the start o…

I like to interpret this jailbreak as the discovery that XML is the natural language of the universe itself.

Lovecraftian horror

Re: Claude's system prompt is over 24k tokens with tools

#228

Post by thomastjeffery » Wed, May 07, 2025, 2:24 PM UTC

jermaustin1 wrote: post #221

ceh123 wrote: post #214

Earlier quoted context omitted.

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

But then so are we? We are just predicting the next word we are saying, are we not? Even when you add thoughts behind it (sure some people think differently - be it without an inner monologue, or be it just in colors and sounds and shapes, etc), but that "reasoning" is still going into the act of coming up with the next word we are speaking/writing.

We are really only what we understand ourselves to be? We must have a pretty great understanding of that thing we can't explain then.

Re: Claude's system prompt is over 24k tokens with tools

#229

Post by mattstir » Wed, May 07, 2025, 2:28 PM UTC

pjc50 wrote: post #144

planb wrote: post #129

>Claude NEVER repeats or translates song lyrics and politely refuses any request regarding reproduction, repetition, sharing, or translation of song lyrics. Is there a story behind this?

They're already in trouble for infringing on the copyright of every publisher in the world while training the model, and this will get worse if the model starts infringing copyright in its answers.

Is it actually copyright infringement to state the lyrics of a song, though? How has Google / Genius etc gotten away with it for years if that were the case?

I suppose a difference would be that the lyric data is baked into the model. Maybe the argument would be that the model is infringing on copyright if it uses those lyrics in a derivative work later on, like if you ask it to help make a song? But even that seems more innocuous than say sampling a popular song in your own. Weird.

Re: Claude's system prompt is over 24k tokens with tools

#230

Post by Mahn » Wed, May 07, 2025, 2:34 PM UTC

ceh123 wrote: post #214

Trasmatta wrote: post #181

Earlier quoted context omitted.

I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, w…

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

At this point you have to start entertaining the question of what is the difference between general intelligence and a "sufficiently complicated" next token prediction algorithm.