Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Anthropic publishes the 'system prompts' that make Claude tick

techcrunch.com

Post reply on HN
Print view View original

151–160 of 290 posts

Re: Anthropic publishes the 'system prompts' that make Claude tick

#151

Post by adwn »

mplewis wrote: post #143
adwn wrote: post #47

Earlier quoted context omitted.

> nor do they respond "Sorry, you're right, 1+1=3, my mistake" without some discernible reason. Look up the Asch conformity experiment [1]. Quite a few people will actually give in to "1+1=3" if all the other people in the room say so. It's not exactly the same as LLM hallucinations, but humans aren't completely immune to this phenomenon. [1] https://en.wikipedia.org/wiki/Asch_conformity_experiments#Me...

It’s not like the circumstances of the experiment are significant to the subjects. You’re a college student getting paid $20 to answer questions for an hour. Your response has no bearing on your pay. Who cares what you say?

> Your response has no bearing on your pay. Who cares what you say?

Then why not say what you know is right?

Re: Anthropic publishes the 'system prompts' that make Claude tick

#152

Post by moffkalast »

Terr_ wrote: post #117
ben_w wrote: post #77

Earlier quoted context omitted.

That's like saying I don't understand what vanilla flavour means just because I can't tell you how many hydrogen atoms vanillin contains — my sense of smell just doesn't do that, and an LLM just isn't normally tokenised in a way to count letters. What I can do, is google it. And an LLM trained on an appropriate source that creates a mapping from nearly-a-whole-word tokens into letter-tokens, that model can (in princi…

> That's like saying I don't understand what vanilla flavour means just because I can't tell you how many hydrogen atoms vanillin contains You're right that there are different kinds of tasks, but there's an important difference here: We probably didn't just have an exchange where you quoted a whole bunch of organic-chemistry details, answered "Yes" when I asked if you were capable of counting the hydrogen atoms, and…

Well there are several problems that lead to the failure.

One is conditioning, models are not typically tuned to say no when they don't know, because confidently bullshitting unfortunately sometimes results in higher benchmark performance which looks good on competitor comparison reports. If you want to see a model that is tuned to do this slightly better than average, see Claude Opus.

Two, you're asking the model to do something that doesn't make any sense to it, since it can't see the letters. It has never seen them, it hasn't learned to intuitively understand what they are. It can tell you what a letter is the same way it can tell you that an old man has white hair despite having no concept of what either of that looks like.

Three, the model is incredibly dumb in terms of raw inteligence, like a third of average human reasoning inteligence for SOTA models at best according to some attempts to test with really tricky logic puzzles that push responses out of the learned distribution. Good memorization helps obfuscate this in lots of cases, especially for 70B+ sized models.

Four, models can only really do an analogue of what "fast thinking" would be in humans, chain of thought and various hidden thought tag approaches help a bit but fundamentally they can't really stop and reflect recursively. So if it knows something it blurts it out, otherwise bullshit it is.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#153

Post by lagniappe »

generalizations wrote: post #123

Claude has been pretty great. I stood up an 'auto-script-writer' recently, that iteratively sends a python script + prompt + test results to either GPT4 or Claude, takes the output as a script, runs tests on that, and sends those results back for another loop. (Usually took about 10-20 loops to get it right) After "writing" about 5-6 python scripts this way, it became pretty clear that Claude is far, far better - if…

That's pretty cool, can I take a look at that? If not, it's okay, just curious.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#154

Post by axus »

creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

I was thinking about LLMs hallucinating function names when writing programs, it's not a bad thing as long as it follows up and generates the code for each function name that isn't real yet. So hallucination is good for purely creative activities, and bad for analyzing the past.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#155

Post by xienze »

creatonez wrote: post #134

Notably, this prompt is making "hallucinations" an officially recognized phenomenon: > If Claude is asked about a very obscure person, object, or topic, i.e. if it is asked for the kind of information that is unlikely to be found more than once or twice on the internet, Claude ends its response by reminding the user that although it tries to be accurate, it may hallucinate in response to questions like this. It uses…

> Probably for the best that users see the words "Sorry, I hallucinated" every now and then.

Wouldn’t “sorry, I don’t know how to answer the question” be better?

Re: Anthropic publishes the 'system prompts' that make Claude tick

#157

Post by fshbbdssbbgdd »

criddell wrote: post #78
Terr_ wrote: post #29

Earlier quoted context omitted.

LLM Prompt Engineering: Injecting your own arbitrary data into a what is ultimately an undifferentiated input stream of word-tokens from no particular source, hoping your sequence will be most influential in the dream-generator output, compared to a sequence placed there by another person, or a sequence that they indirectly caused the system to emit that then got injected back into itself. Then play whack-a-mole unti…

It probably shouldn't be called prompt engineering , even informally. The work of an engineer shouldn't require hope .

I don’t think the people who engineered the Golden Gate Bridge, Apollo 7, or the transistor would have succeeded if they didn’t have hope.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#158

Post by ben_w »

moffkalast wrote: post #152
Terr_ wrote: post #117

Earlier quoted context omitted.

> That's like saying I don't understand what vanilla flavour means just because I can't tell you how many hydrogen atoms vanillin contains You're right that there are different kinds of tasks, but there's an important difference here: We probably didn't just have an exchange where you quoted a whole bunch of organic-chemistry details, answered "Yes" when I asked if you were capable of counting the hydrogen atoms, and…

Well there are several problems that lead to the failure. One is conditioning, models are not typically tuned to say no when they don't know, because confidently bullshitting unfortunately sometimes results in higher benchmark performance which looks good on competitor comparison reports. If you want to see a model that is tuned to do this slightly better than average, see Claude Opus. Two, you're asking the model to…

> because confidently bullshitting unfortunately sometimes results in higher benchmark performance which looks good on competitor comparison reports

You've just reminded me that this was even a recommended strategy in some of the multiple choice tests during my education. Random guessing was scored equally as if you hadn't answered at all

If you really didn't know an answer then every option was equally likely and no benefit, but if you could eliminate just one answer then your expected score from guessing between the others was worthwhile.

Re: Anthropic publishes the 'system prompts' that make Claude tick

#159

Post by manuelmoreale »

ImHereToVote wrote: post #140
ljlolel wrote: post #58

Earlier quoted context omitted.

same as with asking humans to do something

When we do prompt engineering for humans, we use the term Public Relations.

There’s also Social Engineering but I guess that’s a different thing :)

Re: Anthropic publishes the 'system prompts' that make Claude tick

#160

Post by pegasus »

cma wrote: post #99
tritiy wrote: post #33

Earlier quoted context omitted.

My guess is the following: Every time you talk with the LLM it starts with random 'state' (working weights) and then it reads the input tokens and predicts the followup. If you were to save the 'state' (intermediate weights) after inputing the prompt but before inputing user input your would be getting the same output of the network which might have a bias or similar which you have now just 'baked in' into the model.…

No, any randomness is from the temperature setting that just tells mainly tells how much to sample the probability mass of the next output vs choose the exact next most likely (which tends to make them get in repetitive loop like convos).

There's randomness besides what's implied by the temperature. Even when temperature is set to zero, the models are still nondeterministic.
Post reply on HN