Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Claude's system prompt is over 24k tokens with tools

github.com

Post reply on HN
Print view View original

281–290 of 350 posts

Re: Claude's system prompt is over 24k tokens with tools

#281

Post by BobaFloutist »

jermaustin1 wrote: post #221
ceh123 wrote: post #214

Earlier quoted context omitted.

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

But then so are we? We are just predicting the next word we are saying, are we not? Even when you add thoughts behind it (sure some people think differently - be it without an inner monologue, or be it just in colors and sounds and shapes, etc), but that "reasoning" is still going into the act of coming up with the next word we are speaking/writing.

We're not predicting the next word we're most likely to say, we're actively choosing the word that we believe most successfully conveys what we want to communicate. This relies on a theory of mind of those around us and an intentionality of speech that aren't even remotely the same as "guessing what we would say if only we said it"

Re: Claude's system prompt is over 24k tokens with tools

#282

Post by dontlikeyoueith »

Trasmatta wrote: post #181
clysm wrote: post #138

Earlier quoted context omitted.

No, it’s not a threshold. It’s just how the tech works. It’s a next letter guesser. Put in a different set of letters to start, and it’ll guess the next letters differently.

I think we need to start moving away from this explanation, because the truth is more complex. Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token. https://www.anthropic.com/research/tracing-thoughts-language... > Instead, we found that Claude plans ahead. Before starting the second line, it began "thinking" of potential on-topic words that would rhyme with "grab it". Then, w…

> Anthropic's own research showed that Claude does actually "plan ahead", beyond the next token.

For a very vacuous sense of "plan ahead", sure.

By that logic, a basic Markov-chain with beam search plans ahead too.

Re: Claude's system prompt is over 24k tokens with tools

#283

Post by knowaveragejoe »

nullbyte808 wrote: post #182
freehorse wrote: post #95

I was a bit skeptical, so I asked the model through the claude.ai interface "who is the president of the United States" and its answer style is almost identical to the prompt linked https://claude.ai/share/ea4aa490-e29e-45a1-b157-9acf56eb7f8a Meanwhile, I also asked the same to sonnet 3.7 through an API-based interface 5 times, and every time it hallucinated that Kamala Harris is the president (as it should not "know…

I wonder if It could really think if it would be disappointed that Trump won. He was the most illogical and harmfull canidate according to 99% of media.

That would be the takeaway from all of the information available.

Re: Claude's system prompt is over 24k tokens with tools

#284

Post by dontlikeyoueith »

Mahn wrote: post #230
ceh123 wrote: post #214

Earlier quoted context omitted.

I'm not sure if this really says the truth is more complex? It is still doing next-token prediction, but it's prediction method is sufficiently complicated in terms of conditional probabilities that it recognizes that if you need to rhyme, you need to get to some future state, which then impacts the probabilities of the intermediate states. At least in my view it's still inherently a next-token predictor, just with r…

At this point you have to start entertaining the question of what is the difference between general intelligence and a "sufficiently complicated" next token prediction algorithm.

A sufficiently large lookup table in DB is mathematically indistinguishable from a sufficiently complicated next token prediction algorithm is mathematically indistinguishable from general intelligence.

All that means is that treating something as a black box doesn't tell you anything about what's inside the box.

Re: Claude's system prompt is over 24k tokens with tools

#285

Post by morsecodist »

nonethewiser wrote: post #123

For some reason, it's still amazing to me that the model creators means of controlling the model are just prompts as well. This just feels like a significant threshold. Not saying this makes it AGI (obviously its not AGI), but it feels like it makes it something . Imagine if you created a web api and the only way you could modify the responses to the different endpoints are not from editing the code but by sending a…

For me it's the opposite. We don't really have a reliable way of getting the models to do what we want or even to measure if they are doing what we want.

Re: Claude's system prompt is over 24k tokens with tools

#286

Post by int_19h »

manojlds wrote: post #249
lordgilman wrote: post #202

Earlier quoted context omitted.

I like to interpret this jailbreak as the discovery that XML is the natural language of the universe itself.

Isn't Claude trained to work better with XML tags

All modern LLMs seem to prefer XML to other structured markup. It might be because there's so much HTML in the training set, or because it has more redundancy baked in which makes it easier for models to parse.

Re: Claude's system prompt is over 24k tokens with tools

#287

Post by thaumasiotes »

ytpete wrote: post #271
throwawaystress wrote: post #160

Earlier quoted context omitted.

I like the thought, but I don’t think that logic holds generally. I can’t just declare I am someone (or represent someone) without some kind of evidence. If someone just accepted my statement without proof, they wouldn’t have done their due diligence.

Right, imagine if other businesses like banks tried to use a defense like that! "No, it's not my fault some rando cleaned out your bank account because they said they were you."

Imagine?

> This week brought an announcement from a banking association that “identity fraud” is soaring to new levels, with 89,000 cases reported in the first six months of 2017 and 56% of all fraud reported by its members now classed as “identity fraud”.

> So what is “identity fraud”? The announcement helpfully clarifies the concept:

> “The vast majority of identity fraud happens when a fraudster pretends to be an innocent individual to buy a product or take out a loan in their name.

> Now back when I worked in banking, if someone went to Barclays, pretended to be me, borrowed £10,000 and legged it, that was “impersonation”, and it was the bank’s money that had been stolen, not my identity. How did things change?

https://www.lightbluetouchpaper.org/2017/08/26/is-the-city-f...

Re: Claude's system prompt is over 24k tokens with tools

#288

Post by joquarky »

int_19h wrote: post #286
manojlds wrote: post #249

Earlier quoted context omitted.

Isn't Claude trained to work better with XML tags

All modern LLMs seem to prefer XML to other structured markup. It might be because there's so much HTML in the training set, or because it has more redundancy baked in which makes it easier for models to parse.

This is especially efficient when you have multiple pieces of content. You can encapsulate each piece of content into distinct arbitrary XML elements and then refer to them later in your prompt by the arbitrary tag.

Re: Claude's system prompt is over 24k tokens with tools

#290

Post by olddustytrail »

nullbyte808 wrote: post #182
freehorse wrote: post #95

I was a bit skeptical, so I asked the model through the claude.ai interface "who is the president of the United States" and its answer style is almost identical to the prompt linked https://claude.ai/share/ea4aa490-e29e-45a1-b157-9acf56eb7f8a Meanwhile, I also asked the same to sonnet 3.7 through an API-based interface 5 times, and every time it hallucinated that Kamala Harris is the president (as it should not "know…

I wonder if It could really think if it would be disappointed that Trump won. He was the most illogical and harmfull canidate according to 99% of media.

Why wonder when you can just ask it? Tell it to imagine an alternative reality where Trump won.
Post reply on HN