Meanwhile in Estonia, they just agreed to resolve child support disputes using AI... https://www.err.ee/1609701615/pakosta-enamiku-elatisvaidlust...
Positional preferences, order effects, prompt sensitivity undermine AI judgments
31–40 of 87 posts
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#32[flagged]
Ok, but please don't post unsubstantive comments to Hacker News.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#33LLMs are good at discovery, since they know a lot, and can retrieve that knowledge from a query that simpler (e.g. regex-based) search engines with the same knowledge couldn't. For example, an LLM that is input a case may discover an obscure law, or notice a pattern in past court cases which establishes precedent. So they can be helpful to a real judge.
Of course, the judge must check that the law or precedent aren't hallucinated, and apply to the case in the way the LLM claims. They should also prompt other LLMs and use their own knowledge in case the cited law/precedent contradicts others.
There's a similar argument for scientists, mathematicians, doctors, investors, and other fields. LLMs are good at discovery but must be checked.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#34I’d argue real judges are unreliable as well. The real question for me is: are they less reliable than human judges? Probably yes. But I favor a relative measurement to humans than a plain statement like that.
> The real question for me is: are they less reliable than human judges? I've spent some time poking at this. I can't go into details, but the short answer is, "Sometimes yes, sometimes no, and it depends A LOT on how you define 'reliable'." My sense is that, the more boring, mechanical and closed-ended the task is, the more likely an LLM is to be more reliable than a human. Because an LLM is an unthinking machine. I…
> Because an LLM is an unthinking machine.
Unthinking can be pretty powerful these days.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#35Fully agree, I've found that LLMs aren't good at tasks that require evaluation. Think about it, if they were good at evaluation, you could remove all humans in the loop and have recursively self improving AGI. Nice to see an article that makes a more concrete case.
Humans aren't good at validation either. We need tools, experiments, labs. Unproven ideas are a dime a dozen. Remember the hoopla about room temperature superconductivity? The real source of validation is external consequences.
Human experts set the benchmarks and LLM’s cannot match them in most (maybe any?) fields requiring sophisticated judgment.
They are very useful for some things, but sophisticated judgment is not one of them.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#36Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#37Earlier quoted context omitted.
I think the main difference is an AI judge may provide three different rulings if you just ask it the same thing three times. A human judge is much less likely to be so "flip-floppy". You can observe this using any of the present-day LLM's - ask it an architectural/design question, provide it with your thoughts, reasoning, constraints, etc... and see what it tells you. Then... click the "Retry" button and see how sim…
Humans flip-flop all the time. This is a major reason why the Meyers-Briggs Type Indicator does such a poor job of assigning the same person the same Meyers-Briggs type on successive tests. It can be difficult to observe this fact in practice because, unlike for an LLM, you can't just ask a human the exact same question three times in five seconds and get three different answers, because unlike an LLM we have memory.…
We're not talking about labeling data though - we're talking about understanding case law, statutory law, facts, balancing conflicting opinions, arguments, a judge's preconceived notions, experiences, beliefs etc. - many of which are assembled over an entire career.
Those things, I'd argue, are far less likely to change if you ask the same judge over and over. I think you can observe this in reality by considering people's political opinions - which can drift over time but typically remain similar for long durations (or a lifetime).
In real life, we usually don't ask the same judge to remake a ruling over and over - our closest analog is probably a judge's ruling/opinion history, which doesn't change nearly as much as an LLM's "opinion" on something. This is how we label SCOTUS Justices, for example, as "Originalist", etc.
Also, unlike a human, you can radically change an LLM's output by just ever-so-slightly altering the input. While humans aren't above changing their mind based on new facts, they are unlikely to take an opposite position just because you reworded your same argument.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#38LLMs make impressive graders-of-convenience, but their judgments swing wildly with prompt phrasing and option order. Treat them like noisy crowd-raters: randomize inputs, ensemble outputs, and keep a human in the loop whenever single-digit accuracy points matter.
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#39I’d argue real judges are unreliable as well. The real question for me is: are they less reliable than human judges? Probably yes. But I favor a relative measurement to humans than a plain statement like that.
Judges can reason according to principles, and explain this reasoning. LLMs cannot (but they can pretend to, and this pretend chain-of-thought can be marketed as "reasoning"!; see https://news.ycombinator.com/item?id=44069991)
Re: Positional preferences, order effects, prompt sensitivity undermine AI judgments
#40I've done experiments and basically what I found was that LLM models are extremely sensitive to .....language. Well, duh but let me explain a bit. They will give a different quality/accuracy of answer depending on the system prompt order, language use, length, how detailed the examples are, etc... basically every variable you can think of is responsible for either improving or causing detrimental behavior in the output. And it makes sense once you really grok that LLM;s "reason and think" in tokens. They have no internal world representation. Tokens are the raw layer on which they operate. For example if you ask a bilingual human what their favorite color is, the answer will be that color regardless of what language they used to answer that question. For an LLM, that answer might change depending on the language used, because its all statistical data distribution of tokens in training that conditions the response. Anyway i don't want to make a long post here. The good news out of this is that once you have found the best way in asking questions of your model, you can consistently get accurate responses, the trick is to find the best way to communicate with that particular LLM. That's why i am hard at work on making an auto calibration system that runs through a barrage of ways in finding the best system prompts and other hyperparameters for that specific LLM. The process can be fully automated, just need to set it all up.