Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

People are just as bad as my LLMs

wilsoniumite.com

Post reply on HN
Print view View original

21–30 of 173 posts

Re: People are just as bad as my LLMs

#21

Post by robwwilliams »

tehsauce wrote: post #3

There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a prompt like: "Choose a random number", the base pretrained model can give relatively random answers, but after fine tuning to produce responses humans like, they become very biased towards responding with numbers like "7" or "42".

I assume 42 is a joke from deep history and The Hitchhiker’s Guide. Pretty amusing to read the Wikipedia entry:

https://en.wikipedia.org/wiki/42_(number)

Re: People are just as bad as my LLMs

#22

Post by absolutelastone »

moffkalast wrote: post #15
tehsauce wrote: post #3

There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a prompt like: "Choose a random number", the base pretrained model can give relatively random answers, but after fine tuning to produce responses humans like, they become very biased towards responding with numbers like "7" or "42".

It's very funny that people hold the autoregressive nature of LLMs against them, while being far more hardline autoregressive themselves. It's just not consciously obvious.

I think people tend to just not understand what autoregressive methods are capable of doing generally (i.e., basically anything an alternative method can do), and worse they sort of mentally view it as equivalent to a context length of 1.

Re: People are just as bad as my LLMs

#25

Post by foota »

da_chicken wrote: post #13
aidos wrote: post #8

Earlier quoted context omitted.

Why is that ? Whenever I’m giving examples I almost always use 7, something ending in a 7 or something in the 70s

The theory I've heard is that the more prime a number is, the more random it feels. 13 feels more awkward and weird, and it doesn't come up naturally as often as 2 or 3 do in everyday life. It's rare, so it must be more random! I'll give you the most random number I can think of! People tend to avoid extremes, too. If you ask for a number between 1 and 10, people tend to pick something in the middle. Somehow, the ord…

Okay, this is a nitpick, but I don't think ordinal can be used in that way. "Somehow, the ordinal values of the range seem less likely". I'd probably go with extremes of the range? Or endpoints?

Re: People are just as bad as my LLMs

#26

Post by thechao »

tehsauce wrote: post #3

There has been some good research published on this topic of how RLHF, ie aligning to human preferences easily introduces mode collapse and bias into models. For example, with a prompt like: "Choose a random number", the base pretrained model can give relatively random answers, but after fine tuning to produce responses humans like, they become very biased towards responding with numbers like "7" or "42".

Which is weird, because I thought we'd all agreed that the random number was 4?

https://xkcd.com/221/

Re: People are just as bad as my LLMs

#28

Post by fpgaminer »

markbergz wrote: post #19

For anyone interested in these LLM pairwise sorting problems, check out this paper: https://arxiv.org/abs/2306.17563 The authors discuss the person 1 / doc 1 bias and the need to always evaluate each pair of items twice. If you want to play around with this method there is a nice python tool here: https://github.com/vagos/llm-sort

The paper basically sums to suggesting (and analyzing) these otpions:

* Comparing all possible pair permutations eliminates any bias since all pairs are compared both ways, but is exceedingly computationally expensive. * Using a sorting algorithm such as Quicksort and Heapsort is more computationally efficient, and in practice doesn't seem to suffer much from bias. * Sliding window sorting has the lowest computation requirement, but is mildly biased.

The paper doesn't seem to do any exploration of the prompt and whether it has any impact on the input ordering bias. I think that would be nice to know. Maybe assigning the options random names instead of ordinals would reduce the bias. That said, I doubt there's some magic prompt that will reduce the bias to 0. So we're definitely stuck with the options above until the LLM itself gets debiased correctly.

Re: People are just as bad as my LLMs

#29

Post by Jensson »

d4mi3n wrote: post #11
aidos wrote: post #8

Earlier quoted context omitted.

Why is that ? Whenever I’m giving examples I almost always use 7, something ending in a 7 or something in the 70s

My guess is that we bias towards numbers with cultural or personal significance. 7 is lucky in western cultures and is religiously significant (see https://en.wikipedia.org/wiki/7#Culture ). 42 is culturally significant in science fiction, though that's a lot more recent. There are probably other examples, but I imagine the mean converges on numbers with multiple cultural touchpoints.

I have never heard of 7 being a lucky number in western culture and your link doesn't support that. 3 is a lucky number, 13 is an unlucky number, 7 is nothing to me.

So I don't think its that, 7 is still a very common "random number" here even though there is no special cultural significance to it.

Re: People are just as bad as my LLMs

#30

Post by kbenson »

Jensson wrote: post #29
d4mi3n wrote: post #11

Earlier quoted context omitted.

My guess is that we bias towards numbers with cultural or personal significance. 7 is lucky in western cultures and is religiously significant (see https://en.wikipedia.org/wiki/7#Culture ). 42 is culturally significant in science fiction, though that's a lot more recent. There are probably other examples, but I imagine the mean converges on numbers with multiple cultural touchpoints.

I have never heard of 7 being a lucky number in western culture and your link doesn't support that. 3 is a lucky number, 13 is an unlucky number, 7 is nothing to me. So I don't think its that, 7 is still a very common "random number" here even though there is no special cultural significance to it.

It's definitely used in slot machines as a lucky number. Which came first I'm not sure (but I suspect from a sibling comment in the same thread it's based on perceived commonality and primeness historically and became "lucky" in the past because of that).
Post reply on HN