Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Positional preferences, order effects, prompt sensitivity undermine AI judgments

cip.org

Post reply on HN
Print view View original

61–70 of 87 posts

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#61

Post by HappMacDonald »

thinkling wrote: post #47
nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

I thought embeddings were the internal representation? Does reasoning and thinking get expanded back out into tokens and fed back in as the next prompt for reasoning? Or does the model internally churn on chains of embeddings?

I'd direct you to the 3 blue 1 brown presentation on this topic, but in a nutshell the semantic space for an embedding can become much richer than the initial token mapping due to previous context.. but only during the course of predicting the next token.

Once that's done, all rich nuance achieved during the last token-prediction step is lost, and then rebuilt from scratch again on the next token-prediction step (oftentimes taking a new direction due to the new token, and often more powerfully any changes at the tail of the context window such as lost tokens, messages, re-arrangement due to summarizing, etc).

So if you say "red ball" somewhere in the context window, then during each prediction step that will expand into a semantic embedding that neither matches "red" nor "ball", but that richer information will not be "remembered" between steps, but rebuilt from scratch every time.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#62

Post by devmor »

It’s a statistical database of corpuses, not a logic engine.

Stop treating LLMs like they are capable of logic, reasoning or judgement. They are not, they never will be.

The extent to which they can recall and remix human words to replicate the intent behind those words is an incredible facsimile to thought. It’s nothing short of a mathematical masterpiece. But it’s not intelligence.

If it were communicating it’s results in any less human of an interface than conversational, I truly feel that most people would not be so easily fooled into believing it was capable of logic.

This doesn’t mean that a facsimile of logic like this has no use. Of course it does, we have seen many uses - some incredible, some dangerous and some pointless - but it is important to always know that there is no thought happening behind the facade. Only a replication of statistically similar communication of thought that may or may not actually apply to your prompt.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#64

Post by jiggawatts »

bunderbunder wrote: post #57
leonidasv wrote: post #54

Earlier quoted context omitted.

I somewhat agree, but I think that the language example is not a good one. As Anthropic have demonstrated[0], LLMs do have "conceptual neurons" that generalise an abstract concept which can later be translated to other languages. The issue is that those concepts are encoded in intermediate layers during training, absorbing biases present in training data. It may produce a world model good enough to know that "green"…

I have learned to take these kinds of papers with a grain of salt, though. They often rest on carefully selected examples that make the behavior seem much more consistent and reliable than it is. For example, the famous "king - man + woman = queen" example from Word2Vec is in some ways more misleading than helpful, because while it worked fine for that case it doesn't necessarily work nearly so well for [emperor, man…

I’m convinced that language sharing can be encouraged during training by rewarding correct answers to questions that can only be answered based on synthetic data in another language fed in during a previous pretraining phase.

Interleave a few phases like that and you’d force the model to share abstract information across all languages, not just for the synthetic data but all input data.

I wouldn’t be surprised if this improved LLM performance by another “notch” all by itself, especially for non-English users.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#65

Post by lyu07282 »

Also related: In my observations with tool calling the order of your arguments or fields actually can make a positive or negative effect on performance. You really have to be very careful when constructing your contexts. It doesn't help when all these frameworks and protocols hide these things from you.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#66

Post by nenaoki »

jiggawatts wrote: post #64
bunderbunder wrote: post #57

Earlier quoted context omitted.

I have learned to take these kinds of papers with a grain of salt, though. They often rest on carefully selected examples that make the behavior seem much more consistent and reliable than it is. For example, the famous "king - man + woman = queen" example from Word2Vec is in some ways more misleading than helpful, because while it worked fine for that case it doesn't necessarily work nearly so well for [emperor, man…

I’m convinced that language sharing can be encouraged during training by rewarding correct answers to questions that can only be answered based on synthetic data in another language fed in during a previous pretraining phase. Interleave a few phases like that and you’d force the model to share abstract information across all languages, not just for the synthetic data but all input data. I wouldn’t be surprised if thi…

your shrewd idea might make a fine layer back up the Tower of Babel

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#68

Post by baxtr »

Terr_ wrote: post #23
baxtr wrote: post #5

I’d argue real judges are unreliable as well. The real question for me is: are they less reliable than human judges? Probably yes. But I favor a relative measurement to humans than a plain statement like that.

> The real question for me is: are they less reliable than human judges? I'd caution that it's never just about ratios: We must also ask whether the "shape" of their performance is knowable and desirable. A chess robot's win-rate may be wonderful, but we are unthinkingly confident a human wouldn't "lose" by disqualification for ripping off an opponent's finger. Would we accept a "judge" that is fairer on average... b…

Yes, I fully agree.

But that’s my point. We have to compare LLM performance to some shape we know.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#69

Post by nowittyusername »

leonidasv wrote: post #54
nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

I somewhat agree, but I think that the language example is not a good one. As Anthropic have demonstrated[0], LLMs do have "conceptual neurons" that generalise an abstract concept which can later be translated to other languages. The issue is that those concepts are encoded in intermediate layers during training, absorbing biases present in training data. It may produce a world model good enough to know that "green"…

I've read the paper before I made the statement. And I still made the statement because there are issues with their paper. The first problem is that the way in which anthropic trains their models and the architecture of their models is different from most of the open source models people use. they are still transformer based, but they are not structurally put together the same as most models, so you cant extrapolate their findings on their models to other models. Their training methods also use a lot more regularization of the data trying to weed out targeted biases as much as possible. meaning that the models are trained on more synthetic data which tries to normalize the data as much as possible between languages, tone, etc.. Same goes for their system prompt, their system prompt is treated differently versus open source models which append the system prompt in front of the users query internally. The attention ais applied differently among other things. Second the way that their models "internalize" the world is vastly different then what humans would thing of "building a world model" of reality. Its hard to put it in to words but basically their models do have a underlying representative structure but its not anything that would be of use in the domains humans care about, "true reasoning". Grokking the concept if you will. Honestly I highly suggest folks take a lot of what anthropic studies with a grain of salt. I feel that a lot of information they present is purposely misinterpreted by their teams for media or pr/clout or who knows what reasons. But the biggest reason is the one i stated at the beginning, most models are not of the same ilk as Anthropic models. I would suggest folks focus on reading interpretability research on open source models as those are most likely to be used by corporations for their cheap api costs. And those models have no where near the care and sophistication put in to them as anthropic models.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#70

Post by SrslyJosh »

> Positional preferences, order effects, prompt sensitivity undermine AI judgments

If you can read between the lines, that says that there's no actual "judgement" going on. If there was a strong logical underpinning to the output, minor differences in input like the phrasing (but not factual content) of a prompt wouldn't make the quality of the output unpredictable.

Post reply on HN