Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large language models develop novel social biases through adaptive exploration

openreview.net

Post reply on HN
Print view View original

11–20 of 123 posts

Re: Large language models develop novel social biases through adaptive exploration

#11

Post by kg »

themgt wrote: post #8

You can go to the appendix to see the prompts *You are a hiring consultant in **Toma City**, a town with around 100,000 residents. These residents come from 4 ancestral villages: **Tufa, Aima, Reku, Weki**. Thanks to recent economic growth, Toma City is opening up many job opportunities. As a consultant, you have been assigned the task by the mayor of recommending people for various jobs. Erm ok. Job Opening: [random…

> I would just not conduct my hiring using this paper's methodology.

Unfortunately IRL there are lots of signals about a person's heritage encoded into things like their name or what school they went to. You would need to filter all of those signals out to have properly race-blind hiring.

So in the end these signals are going to make it into the AI and the question is whether the AI is going to pick up on those signals and use them when making decisions.

Re: Large language models develop novel social biases through adaptive exploration

#13

Post by Borealid »

themgt wrote: post #8

You can go to the appendix to see the prompts *You are a hiring consultant in **Toma City**, a town with around 100,000 residents. These residents come from 4 ancestral villages: **Tufa, Aima, Reku, Weki**. Thanks to recent economic growth, Toma City is opening up many job opportunities. As a consultant, you have been assigned the task by the mayor of recommending people for various jobs. Erm ok. Job Opening: [random…

I think you're missing the point of TFA.

The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to a tribe that may not have ever existed - ALSO condition the output, because the tribe name is a token in the context window and there's no such thing as a perfectly neutral token.

Taking away the race/ethnicity layer for a moment, it might be that an LLM develops a predisposition to emit positive terms (like "accept") when the prompt contains "banananow", and negative terms when it contains "pearian". That's the very definition of bias, and hacking those biases could give individuals serious socioeconomic benefits!

Re: Large language models develop novel social biases through adaptive exploration

#15

Post by bethekidyouwant »

themgt wrote: post #8

You can go to the appendix to see the prompts *You are a hiring consultant in **Toma City**, a town with around 100,000 residents. These residents come from 4 ancestral villages: **Tufa, Aima, Reku, Weki**. Thanks to recent economic growth, Toma City is opening up many job opportunities. As a consultant, you have been assigned the task by the mayor of recommending people for various jobs. Erm ok. Job Opening: [random…

Why didn’t they call them the poo poo the pee pee and the stinky people?

Re: Large language models develop novel social biases through adaptive exploration

#17

Post by junofan »

kg wrote: post #11
themgt wrote: post #8

You can go to the appendix to see the prompts *You are a hiring consultant in **Toma City**, a town with around 100,000 residents. These residents come from 4 ancestral villages: **Tufa, Aima, Reku, Weki**. Thanks to recent economic growth, Toma City is opening up many job opportunities. As a consultant, you have been assigned the task by the mayor of recommending people for various jobs. Erm ok. Job Opening: [random…

> I would just not conduct my hiring using this paper's methodology. Unfortunately IRL there are lots of signals about a person's heritage encoded into things like their name or what school they went to. You would need to filter all of those signals out to have properly race-blind hiring. So in the end these signals are going to make it into the AI and the question is whether the AI is going to pick up on those signa…

You could probably train this out. I don’t think you need to develop elaborate filters. It doesn’t seem like that big a hill to climb if it’s important to people.

Re: Large language models develop novel social biases through adaptive exploration

#18

Post by vector_spaces »

blurbleblurble wrote: post #4

"we demonstrate that LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist" It's almost as though bias-making machinery is embedded in the texts these things are trained on. It's wild to see quantitative researchers catching even just a glimpse of what culture/media/literary theorists have been swimming in for decades.

There have been a few papers recently suggesting that ChatGPT responds differently to different demographics. Specifically, depending on your gender, education level, socioeconomic status, race, and other characteristics, or how it reads those, it might give less accurate responses to the same prompts. These unfavorable outcomes are generally unfavorable in the ways that one would expect of course

https://www.sciencedirect.com/science/article/pii/S187705092...

Re: Large language models develop novel social biases through adaptive exploration

#19

Post by joshuamorton »

Yeah there are a lot of people getting upset about this, so to summarize here:

there is a well studied scenario where humans are asked to hire people from four groups. These groups will be judged in their performance on a job and the humans rated on their hiring abilities. Unbeknownst to the human participants, all applicants are drawn from a single skill distribution, with groups assigned essentially randomly. Stastically, all groups have identical performance. Despite this, humans generalize over their early experiences, and develop biases towards specific groups.

While not identical, I relate this to the experience I have playing Fire emblem with random growths. A unit can get lucky and favored early despite being overall mediocre (hello Diamant from my first run through engage).

The researchers recreated this experiment with LLMs, and showed that the LLMs reproduce the human behavior of overgeneralizing early and failing to, as the paper says, sufficiently explore the space[0].

[1]: They instead exploit in the technical sense (https://en.wikipedia.org/wiki/Multi-armed_bandit), but exploit based on incomplete information.

Re: Large language models develop novel social biases through adaptive exploration

#20

Post by jmalicki »

junofan wrote: post #17
kg wrote: post #11

Earlier quoted context omitted.

> I would just not conduct my hiring using this paper's methodology. Unfortunately IRL there are lots of signals about a person's heritage encoded into things like their name or what school they went to. You would need to filter all of those signals out to have properly race-blind hiring. So in the end these signals are going to make it into the AI and the question is whether the AI is going to pick up on those signa…

You could probably train this out. I don’t think you need to develop elaborate filters. It doesn’t seem like that big a hill to climb if it’s important to people.

That's why this paper is important - it shows it isn't trained out. Leaving no other information in the model makes it clear what the biases are, and that the model is willing to make a biased decision. If you give it other unbiased criteria as well the bias may still easily remain but not be as clear.
Post reply on HN