Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic publishes the 'system prompts' that make Claude tick

techcrunch.com

Post reply on HN
Print view View original

161–170 of 290 posts

Re: Anthropic publishes the 'system prompts' that make Claude tick

#161

Post by xscott »

Terr_ wrote: post #37
xscott wrote: post #34

Earlier quoted context omitted.

How do random people you meet in the grocery store measure-up with this standard?

Well, your own mind axiomatically works, and we can safely assume the beings you meet in the grocery store have minds like it which have the same capabilities and operate on cause-and-effect principles that are known (however imperfectly) to medical and psychological science. (If you think those shoppers might be hollow shells controlled by a remote black box, ask your doctor about Capgras Delusion. [0]) Plus they do…

> Well, your own mind axiomatically works

At the risk of teeing-up some insults for you to bat at me, I'm not so sure my mind does that very well. I think the talking jockey on the camel's back analogy is a pretty good fit. The camel goes where it wants, and the jockey just tries to explain it. Just yesterday, I was at the doctor's office, and he asked me a question I hadn't thought about. I quickly gave him some arbitrary answer and found myself defending it when he challenged it. Much later I realized what I wished I had said. People are NOT axiomatic most of the time, and we're not quick at it.

As for ways to make LLMs fail the Turing test, I think these are early days. Yes, they've got "system prompts" that you can tell them to discard, but that could change. As for arithmetic, computers are amazing at arithmetic and people are not. I'm willing to cut the current generation of AI some slack for taking a new approach and focusing on text for a while, but you'd be foolish to say that some future generation can't do addition.

Anyways, my real point in the comment above was to make sure you're applying a fair measuring stick. People (all of us) really aren't that smart. We're monkeys that might be able to do calculus. I honestly don't know how other people think. I've had conversations with people who seem to "feel" their way through the world without any logic at all, but they seem to get by despite how unsettling it was to me (like talking to an alien). Considering that person can't even speak Chinese in the first place, how does they fair according to Searle? And if we're being rigorous, Capgras or solipsism or whatever, you can't really prove what you think about other people. I'm not sure there's been any progress on this since Descartes.

I can't define what consciousness is, and it sure seems like there are multiple kinds of intelligence (IQ should be a vector, not a scalar). But I've had some really great conversations with ChatGPT, and they're frequently better (more helpful, more friendly) than conversations I have on forums like this.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#162

Post by MacsHeadroom »

miki123211 wrote: post #48
zevv wrote: post #28

Earlier quoted context omitted.

It actually still scares the hell out of me that this is the way even the experts 'program' this technology, with all the ambiguities rising from the use of natural language.

Keep in mind that this is not the only way the experts program this technology. There's plenty of fine-tuning and RLHF involved too, that's mostly how "model alignment" works for example. The system prompt exists merely as an extra precaution to reinforce the behaviors learned in RLHF, to explain some subtleties that would be otherwise hard to learn, and to fix little mistakes that remain after fine-tuning. You can v…

Anthropic/Claude does not use any RLHF.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#163

Post by teqsun »

MacsHeadroom wrote: post #162
miki123211 wrote: post #48

Earlier quoted context omitted.

Keep in mind that this is not the only way the experts program this technology. There's plenty of fine-tuning and RLHF involved too, that's mostly how "model alignment" works for example. The system prompt exists merely as an extra precaution to reinforce the behaviors learned in RLHF, to explain some subtleties that would be otherwise hard to learn, and to fix little mistakes that remain after fine-tuning. You can v…

Anthropic/Claude does not use any RLHF.

Is that a claim they've made or has that been externally proven?

Re: Anthropic publishes the 'system prompts' that make Claude tick

#164

Post by creatonez »

hotstickyballs wrote: post #141
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

“Hallucination” has been in the training data much earlier than even llms. The easiest way to control this phenomenon is using the “hallucination” tokens, hence the construction of this prompt. I wouldn’t say that this makes things official.

> The easiest way to control this phenomenon is using the “hallucination” tokens, hence the construction of this prompt.

That's what I'm getting at. Hallucinations are well known about, but admitting that you "hallucinated" in a mundane conversation is a rare thing to happen in the training data, so a minimally prompted/pretrained LLM would be more likely to say "Sorry, I misinterpreted" and then not realize just how grave the original mistake was, leading to further errors. Add the word hallucinate and the chatbot is only going to humanize the mistake by saying "I hallucinated", which lets it recover from extreme errors gracefully. Other words, like "confabulation" or "lie", are likely more prone to causing it to have an existential crisis.

It's mildly interesting that the same words everyone started using to describe strange LLM glitches also ended up being the best token to feed to make it characterize its own LLM glitches. This newer definition of the word is, of course, now being added to various human dictionaries (such as https://en.wiktionary.org/wiki/hallucinate#Verb) which will probably strengthen the connection when the base model is trained on newer data.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#165

Post by SatvikBeri »

xienze wrote: post #155
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

> Probably for the best that users see the words "Sorry, I hallucinated" every now and then. Wouldn’t “sorry, I don’t know how to answer the question” be better?

That requires more confidence. If there's a 50% chance something is true, I'd rather have Claude guess and give a warning than say it doesn't know how to answer.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#166

Post by ForHackernews »

"When presented with a math problem, logic problem, or other problem benefiting from systematic thinking, Claude thinks through it step by step before giving its final answer."

... do AI makers believe this works? Like do think Claude is a conscious thing that can be instructed to "think through" a problem?

All of these prompts (from Anthropic and elsewhere) have a weird level of anthropomorphizing going on. Are AI companies praying to the idols they've made?

Re: Anthropic publishes the 'system prompts' that make Claude tick

#167

Post by armchairhacker »

creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

How can Claude "know" whether something "is unlikely to be found more than once or twice on then internet"? Unless there are other sources that explicitly say "[that thing] is obscure". I don't think LLMs can report if something was encountered more/less often in their training data, there are too many weights and neither us nor them know exactly what each of them represents.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#168

Post by cjbillington »

MacsHeadroom wrote: post #162
miki123211 wrote: post #48

Earlier quoted context omitted.

Keep in mind that this is not the only way the experts program this technology. There's plenty of fine-tuning and RLHF involved too, that's mostly how "model alignment" works for example. The system prompt exists merely as an extra precaution to reinforce the behaviors learned in RLHF, to explain some subtleties that would be otherwise hard to learn, and to fix little mistakes that remain after fine-tuning. You can v…

Anthropic/Claude does not use any RLHF.

What do they do instead? Given we're not talking to a base model.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#169

Post by creatonez »

xienze wrote: post #155
creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

> Probably for the best that users see the words "Sorry, I hallucinated" every now and then. Wouldn’t “sorry, I don’t know how to answer the question” be better?

Not necessarily. The LLM doesn't know what it can answer before it tries to. So in some cases it might be better to make an attempt and then later characterize it as a hallucination, so that the error doesn't spill over and produce even more incoherent nonsense. The chatbot admitting that it "hallucinated" is a strong indication to itself that part of the previous text is literal nonsense and cannot be trusted, and that it needs to take another approach.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#170

Post by gdiamos »

We know that LLMs hallucinate, but we can also remove them.

I’d love to see a future generation of a model that doesn’t hallucinate on key facts that are peer and expert reviewed.

Like the Wikipedia of LLMs

https://arxiv.org/pdf/2406.17642

That’s a paper we wrote digging into why LLMs hallucinate and how to fix it. It turns out to be a technical problem with how the LLM is trained.

Post reply on HN