Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Positional preferences, order effects, prompt sensitivity undermine AI judgments

cip.org

Post reply on HN
Print view View original

41–50 of 87 posts

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#41

Post by pton_xd »

nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

Yep, LLMs tell you "what you want to hear." I can usually predict the response I'll get based on how I phrase the question.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#42

Post by acdha »

Alupis wrote: post #19
baxtr wrote: post #5

I’d argue real judges are unreliable as well. The real question for me is: are they less reliable than human judges? Probably yes. But I favor a relative measurement to humans than a plain statement like that.

I think the main difference is an AI judge may provide three different rulings if you just ask it the same thing three times. A human judge is much less likely to be so "flip-floppy". You can observe this using any of the present-day LLM's - ask it an architectural/design question, provide it with your thoughts, reasoning, constraints, etc... and see what it tells you. Then... click the "Retry" button and see how sim…

I’d think there’s also a key adversarial problem: a human judge has a conceptual understanding and you aren’t going to be able to slightly tweak your wording to get wildly different outcomes the way LLMs are vulnerable to.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#43

Post by giancarlostoro »

batshit_beaver wrote: post #8
giancarlostoro wrote: post #3

I listen to online debates, especially political ones on various platforms, and man. The AI slop that people slap around at each other is beyond horrendous. I would not want an LLM being the final say on something critical. I want the opposite, an LLM should identify things that need follow up review by a qualified person, a person should still confirm the things that "pass" but they can then prioritize what to valid…

I don't even trust LLMs enough to spot content that requires validation or nuance.

LLM's are 100% trust but verify.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#44

Post by NitpickLawyer »

shahbaby wrote: post #11

Fully agree, I've found that LLMs aren't good at tasks that require evaluation. Think about it, if they were good at evaluation, you could remove all humans in the loop and have recursively self improving AGI. Nice to see an article that makes a more concrete case.

I think there's more nuance, and the way I read the article is more "beware of these shortcomings", instead of "aren't good". LLM-based evaluation can be good. Several models have by now been trained on previous-gen models used in filtering data and validating RLHF data (pairwise or even more advanced). LLama3 is a good example of this.

My take from this article is that there are plenty of gotchas along the way, and you need to be very careful in how you structure your data, and how you test your pipelines, and how you make sure your tests are keeping up with new models. But, like it or not, LLM based evaluation is here to stay. So explorations into this space are good, IMO.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#45

Post by bunderbunder »

Alupis wrote: post #37
bunderbunder wrote: post #26

Earlier quoted context omitted.

Humans flip-flop all the time. This is a major reason why the Meyers-Briggs Type Indicator does such a poor job of assigning the same person the same Meyers-Briggs type on successive tests. It can be difficult to observe this fact in practice because, unlike for an LLM, you can't just ask a human the exact same question three times in five seconds and get three different answers, because unlike an LLM we have memory.…

We're not talking about labeling data though - we're talking about understanding case law, statutory law, facts, balancing conflicting opinions, arguments, a judge's preconceived notions, experiences, beliefs etc. - many of which are assembled over an entire career. Those things, I'd argue, are far less likely to change if you ask the same judge over and over. I think you can observe this in reality by considering pe…

I think that that gets back to the whole memory thing. A person is unlikely to forget those kinds of decisions.

But there has been research indicating, for example, that judges' rulings vary with the time of day. In a way that implies that, if it were possible to construct such an experiment, you might find that the same judge given the same case would rule in very different ways depending on whether you present it in the morning or in the afternoon. For example judges tend to hand out significantly harsher penalties toward the end of the work day.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#46

Post by wyett »

nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

[dead]

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#47

Post by thinkling »

nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

I thought embeddings were the internal representation? Does reasoning and thinking get expanded back out into tokens and fed back in as the next prompt for reasoning? Or does the model internally churn on chains of embeddings?

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#49

Post by not_maz »

nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

I found an absolutely fascinating analysis on precisely this topic by an AI researcher who's also a writer: https://archive.ph/jgam4

LLMs can generate convincing editorial letters that give a real sense of having deeply read the work. The problem is that they're extremely sensitive, as you've noticed, to prompting as well as order bias. Present it with two nearly identical versions of the same text, and it will usually choose based on order. And social proof type biases to which we'd hope for machines to be immune can actually trigger 40+ point swings on a 100-point scale.

If you don't mind technical details and occasional swagger, his work is really interesting.

Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments

#50

Post by jonplackett »

pton_xd wrote: post #41
nowittyusername wrote: post #40

I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the outp…

Yep, LLMs tell you "what you want to hear." I can usually predict the response I'll get based on how I phrase the question.

I feel like LLMs have a bit of the Clever Hans effect. It takes a lot of my cues as to what it thinks I want it to say or opinion it thinks I want it to have.

Clever Hans was a horse who people thought could do maths by tapping his hoof. But actually he was just reading the body language of the person asking the question. Noticing them tense up as he got to the right number of stamps and stopping - still pretty smart for a horse, but the human was still doing the maths!

Post reply on HN