Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic publishes the 'system prompts' that make Claude tick

techcrunch.com

Post reply on HN
Print view View original

181–190 of 290 posts

Re: Anthropic publishes the 'system prompts' that make Claude tick

#181

Post by jcims »

ithkuil wrote: post #24
cubefox wrote: post #15

Earlier quoted context omitted.

And "kinda" is an understatement. It understands you very well, perhaps even better than the average human would. (Average humans often don't understand jargon.)

Indeed the understanding part is very good. I just tried this: " I'm dykslegsik I offen Hawe problems wih sreach ennginnes bat eye think yoy wiw undrestand my " Gpt-4o replied: " I understand you perfectly! If you have trouble with search engines or anything else, feel free to ask me directly, and I'll do my best to help you. Just let me know what you're looking for or what you need assistance with! "

I've recently noticed that I've completely stopped fixing typos in my prompts.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#182

Post by furyofantares »

armchairhacker wrote: post #167
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

I think it could be fine tuned to give it an intuition, like how you or I have an intuition about what might be found on the internet.

That said I've never seen it give the response suggested in this prompt and I've tried loads of prompts just like this in my own workflows and they never do anything.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#183

Post by [deleted] »

th0ma5 wrote: post #176
brulard wrote: post #172

Earlier quoted context omitted.

I believe Claude is aware if information close to the one retrieved from the vector space is scarce. I'm no expert, but i imagine it makes a query to the vector database and get the data close enough to places pointed out by the prompt. And it may see that part of the space is quite empty. If this is far off, someone please explain.

I think good, true, but rare information would also fit that definition so it'd be a shame if it discovered something that could save humanity but then discounted it as probably not accurate.

[deleted]

Re: Anthropic publishes the 'system prompts' that make Claude tick

#184

Post by marcus0x62 »

blooalien wrote: post #175
marcus0x62 wrote: post #120

Earlier quoted context omitted.

I understand it to be by predicting the next most likely output token based on previous user input. I also understand that, simplistic though the above explanation is and perhaps is even wrong in some way, it to be a more thorough explanation than anyone thus far has been able to provide about how, exactly, human consciousness and thought works. In any case, my point is this: nobody can say “LLMs don’t reason in the…

Does your brain completely stop doing anything between verbal statements (output)? An LLM does stop doing stuff between requests to generate a string of language tokens (their entire purpose). When not actually generating tokens, an LLM doesn't sit there and think things like "Was what I just said correct?" or "Hmm. That was an interesting discussion. I think I'll go research more on the topic". Nope. It just sits th…

What does that have to do with how the human brain operates while generating a thought as compared to how an LLM generates output? You’ve only managed to state something everyone knows (people think about stuff constantly) without saying anything new about the unknown being discussed (how people think.)

Re: Anthropic publishes the 'system prompts' that make Claude tick

#185

Post by tqi »

cjbillington wrote: post #168
MacsHeadroom wrote: post #162

Earlier quoted context omitted.

Anthropic/Claude does not use any RLHF.

What do they do instead? Given we're not talking to a base model.

Supposedly they use "RLAIF", but honestly given that the first step is to "generate responses... using a helpful-only AI assistant" it kinda sounds like RLHF with more steps.

https://www.anthropic.com/research/constitutional-ai-harmles...

Re: Anthropic publishes the 'system prompts' that make Claude tick

#186

Post by rf15 »

brulard wrote: post #172
armchairhacker wrote: post #167

Earlier quoted context omitted.

How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

I believe Claude is aware if information close to the one retrieved from the vector space is scarce. I'm no expert, but i imagine it makes a query to the vector database and get the data close enough to places pointed out by the prompt. And it may see that part of the space is quite empty. If this is far off, someone please explain.

I wonder if that's the case - the prompt text (like all text interaction with LLMs) is seen from "within" the vector space, while sparcity is only observable from the "outside"

Re: Anthropic publishes the 'system prompts' that make Claude tick

#187

Post by llmfan »

criddell wrote: post #78
Terr_ wrote: post #29

Earlier quoted context omitted.

LLM Prompt Engineering: Injecting your own arbitrary data into a what is ultimately an undifferentiated input stream of word-tokens from no particular source, hoping your sequence will be most influential in the dream-generator output, compared to a sequence placed there by another person, or a sequence that they indirectly caused the system to emit that then got injected back into itself. Then play whack-a-mole unti…

It probably shouldn't be called prompt engineering , even informally. The work of an engineer shouldn't require hope .

It should be called prompt science.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#188

Post by GaggiX »

armchairhacker wrote: post #167
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

I thought the same thing, but when I test the model on like titles of new mangas and stuff that were not present in the training dataset, the model seems to know of not knowing. I wonder if it's a behavior learned during fine-tuning.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#189

Post by viraptor »

armchairhacker wrote: post #167
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

LLMs encode their certainty enough to output it again. They don't need to be specifically trained for this. https://ar5iv.labs.arxiv.org/html/2308.16175

Re: Anthropic publishes the 'system prompts' that make Claude tick

#190

Post by Terr_ »

roughly wrote: post #75
Terr_ wrote: post #32

Earlier quoted context omitted.

Given that it's a big next-word-predictor, I think it has to do with matching the training data. For the vast majority of text out there, someone's personality, goals, etc. are communicated via a narrator describing how thing are. (Plays, stories, almost any kind of retelling or description.) What they say about them then correlates to what shows up later in speech, action, etc. In contrast, it's extremely rare for s…

Many people are telling me the second one is weird. They come up to me and say, “Sir, that thing they’re doing, the things they’re saying, are the weirdest things we’ve ever heard!” And I agree with them. And let me tell you, we’re going to do something about it.

I didn't have that in mind when I wrote the post, and I think my conflicted feelings are best summarized by the idiom: "Thanks, I Hate It."
Post reply on HN