Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Positional preferences, order effects, prompt sensitivity undermine AI judgments

cip.org

Post reply on HN
Print view View original

71–80 of 87 posts

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#71

Post by Wilsoniumite »

SrslyJosh wrote: post #70

> Positional preferences, order effects, prompt sensitivity undermine AI judgments If you can read between the lines, that says that there's no actual "judgement" going on. If there was a strong logical underpinning to the output, minor differences in input like the phrasing (but not factual content) of a prompt wouldn't make the quality of the output unpredictable.

Yes and no. People also exhibit these biases, but because degree matters, and because we have no other choice, we still trust them most of the time. That's to say; bias isn't always completely invalidating. I wrote a slightly satirical piece "People are just as bad as my LLMs" here: https://wilsoniumite.com/2025/03/10/people-are-just-as-bad-a...

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#72

Post by einrealist »

PrimordialEdg71 wrote: post #38

LLMs make impressive graders-of-convenience, but their judgments swing wildly with prompt phrasing and option order. Treat them like noisy crowd-raters: randomize inputs, ensemble outputs, and keep a human in the loop whenever single-digit accuracy points matter.

"and keep a human in the loop whenever single-digit accuracy points matter"

So we are supposed to give up on accuracy now? At least with humans (assuming good actors) I can assume an effort for accuracy and correctness. And I can build trust based on some resume and on former interaction.

With LLMs, this is more like a coin-flip with each prompt. And since the models are updated constantly, its hard to build some sort of resume. In the meantime, people - especially casual users - might just trust outputs, because its convenient. A single digit error is harder to find. The costs of validating outputs increases with increased accuracy of LLMs. And casual users tend to skip validation because "its a computer and computers are correct".

I fear an overall decrease in quality wherever LLMs are included. And any productivity gains are eaten by that.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#74

Post by tiahura »

SrslyJosh wrote: post #70

> Positional preferences, order effects, prompt sensitivity undermine AI judgments If you can read between the lines, that says that there's no actual "judgement" going on. If there was a strong logical underpinning to the output, minor differences in input like the phrasing (but not factual content) of a prompt wouldn't make the quality of the output unpredictable.

Word plinko

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#75

Post by ACCount36 »

SrslyJosh wrote: post #70

> Positional preferences, order effects, prompt sensitivity undermine AI judgments If you can read between the lines, that says that there's no actual "judgement" going on. If there was a strong logical underpinning to the output, minor differences in input like the phrasing (but not factual content) of a prompt wouldn't make the quality of the output unpredictable.

You could say the same about human "judgement" then.

Humans display biases very similar to that of LLMs. This is not a coincidence. LLMs are trained on human-generated data. They attempt to replicate human reasoning - bias and all.

There are decisions where "strong logical underpinning" is strong enough to completely drown out the bias and the noise. And then there are decisions that aren't nearly as clear-cut - allowing the bias and the noise to creep into the outcomes. This is true for human and LLM "judgement" both.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#76

Post by gus_massa »

nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

> For example if you ask a bilingual human what their favorite color is, the answer will be that color regardless of what language they used to answer that question.

It's a very interesting question. Has someone measured it? Bonus point for using a conceal way so the subjects don't realize you care about colors.

Anyway, I don't expect something interesting with colors, but it may be interesting with food (I guess, in particular desserts).

Imagine you live in England and one of your parents is form France and you go there every year to meet your grandparents, and your other parent is from Germany and you go there every year to meet your grandparents. What is your favorite dessert? I guess when you are speaking in one language you are more connected to the memories of the holidays there and the grandparents and you may choose differently.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#77

Post by photochemsyn »

Disappointing that they didn't benchmark DeepSeek side by side with OpenAI and Gemini although thelist of funders for cip may explain that:

https://www.cip.org/funding-partnerships

Incidentally DeepSeek will give very interesting results if you ask it for a tutorial on prompt engineering - be sure to ask it how to effectively use 'attention anchors' to create 'well-structured prompts', and why rambling disorganized prompts are usually, but not always, detrimental, depending on whether you want 'associative leaps' or not.

P.S. I find this intro very useful:

> "Task: evaluate the structure of the following prompt in terms of attention anchors and likelihood of it generating a well-structured response. Do not actually reply to the prompt, all we need is an analysis of the structure. Prompt begins:"

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#78

Post by panstromek »

Good, but really none of this should be surprising, given that LLMs are a giant text statistic that generate text based on that statistic. Quirks of that statistic will show up as quirks of the output.

When you think about it like that, it doesn't really make sense to assume they have some magical correctness properties. In some sense, they don't classify, they immitate what classification looks like.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#79

Post by perching_aix »

panstromek wrote: post #78

Good, but really none of this should be surprising, given that LLMs are a giant text statistic that generate text based on that statistic. Quirks of that statistic will show up as quirks of the output. When you think about it like that, it doesn't really make sense to assume they have some magical correctness properties. In some sense, they don't classify, they immitate what classification looks like.

> In some sense, they don't classify, they immitate what classification looks like.

I thought I've seen it all when people decided to consider AI a marketing term and started going off about how current mainstream AI products aren't """"real AI"""", but this is next level.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#80

Post by mschuster91 »

amlib wrote: post #58
armchairhacker wrote: post #33

LLMs are good at discovery, since they know a lot, and can retrieve that knowledge from a query that simpler (e.g. regex-based) search engines with the same knowledge couldn't. For example, an LLM that is input a case may discover an obscure law, or notice a pattern in past court cases which establishes precedent. So they can be helpful to a real judge. Of course, the judge must check that the law or precedent aren't…

I would add that "hallucinations" aren't even the only failure mode a LLM can have, it can partially or completely miss what its supposed to find in the discovery process and lead you to believe that there just isn't anything worth pursuing in that particular venue.

> it can partially or completely miss what its supposed to find in the discovery process and lead you to believe that there just isn't anything worth pursuing in that particular venue.

The problem is that American and UK legal systems never got forced to prune the sometimes centuries-old garbage. And modern Western legal systems tend to have more explicit laws and regulations instead of prior case law, but still, they also accumulate lots of garbage.

IMHO, all laws and regulations should come with a set expiry date. If the law or regulation is not renewed, it gets dropped off the book. And for legal systems that have case law, court rulings should expire no later than five years, to force a transition to a system where the law-passing body is forced to work for its money.

Post reply on HN