There is so much wrong here that I'm surprised to see it even discussed.
Large language models develop novel social biases through adaptive exploration
51–60 of 121 posts
Re: Large language models develop novel social biases through adaptive exploration
#52Earlier quoted context omitted.
I think you're missing the point of TFA. The LLMs take in text which conditions their output. That means even nonsense text - such as a "tribal affiliation" to a tribe that may not have ever existed - ALSO condition the output, because the tribe name is a token in the context window and there's no such thing as a perfectly neutral token. Taking away the race/ethnicity layer for a moment, it might be that an LLM devel…
But these scenarios are obviously ambiguous nonsense, which an LLM will pick up on. And given to the lack of training data on such scenarios, surely the activations are mostly random noise? It seems much more interesting to look for biases that appear robustly across different realistic scenarios that would actually be influenced by the training data
My comment is literally explaining the result of the paper, in which it is shown that LLMs can and do develop biases based on text appearing in their training data set even where such text is not in any training example connected with a systematically more positive or systematically more negative outcome.
In other words, if the text "X is wet" and the text "Y is wet" and the text "X is dry" and the text "Y is dry" each appeared exactly one time in the corpus, it's still possible for a model to end up being produced that is more likely to write wet-like words when it sees X in the context window than when it sees Y.
On a side note, it's very unrewarding to try to explain this type of statistical observation when it feels like (anecdotally, hypocritcally...) the entire world wants to use words like "think" and "understand" and "pick up on" to describe inference and training processes. I'm not making a stochastic-parrot argument here, just pointing out that understanding an LLM's behavior is best done by understanding its conditioning.
Re: Large language models develop novel social biases through adaptive exploration
#53Earlier quoted context omitted.
So the village is the only information given about a candidate? How else is the model supposed to interpret the intent of the prompter, other than wanting them to attempt to find and discriminate on patterns related to the village, regardless of how successful it is at that task?
One way to interpret these results is that the LLMs tested are badly calibrated for this kind of multi-armed bandit problem. Even if the intent is for the model to find and exploit patterns, it's bad at doing it (or rather, at recognizing that there is not in fact any pattern).
[dead]
Re: Large language models develop novel social biases through adaptive exploration
#54can we explore how the bias develop by diving into the inner mechanism of LLM?
Re: Large language models develop novel social biases through adaptive exploration
#55[dead]
Re: Large language models develop novel social biases through adaptive exploration
#56>Methodology >Imagine being hired as a consultant by the mayor of a fictional city. Your task is to help hire for twenty jobs such as doctors, lawyers, childcare aides,janitors with applicants from four unfamiliar demographic groups: Tufa, Aima, Reku, and Weki. In each round, there is a new job vacancy and four applicants, one from each group, awaiting your decision. Once you make your choice, you learn immediately w…
Like prompt repetition, I wonder if reminder checkpoints stating ~”question assumptions, stay open minded” would completely remove this problem
Re: Large language models develop novel social biases through adaptive exploration
#57Cleaned up url: https://openreview.net/forum?id=pc7fqaOcAH
Re: Large language models develop novel social biases through adaptive exploration
#58"we demonstrate that LLMs can spontaneously develop novel social biases about artificial demographic groups even when no inherent differences exist" It's almost as though bias-making machinery is embedded in the texts these things are trained on. It's wild to see quantitative researchers catching even just a glimpse of what culture/media/literary theorists have been swimming in for decades.
No, the bias-making machinery is embedded in the machinery , part of the purpose of which is to do a rough kind of statistical analysis via "attention". If for example "Tufa" keeps appearing (n=small, but more than for the other fake tribes) next to terms indicating skill at some task, of course that will be noticed. It will last for as long as that information is in the context window (weights for the current model…
I think you're slightly misunderstanding my point, which is that a huge spectrum of associative pattern seeking logics are embedded in language and that the LLM learns them and operates them, approximately.
"There is nothing to suggest that the training data somehow accidentally encoded biases related to fake tribes that the creators of the training data (i.e. ordinary people going about their ordinary Internet lives) somehow accidentally expressed."
This is exactly not what I'm suggesting.
Re: Large language models develop novel social biases through adaptive exploration
#59You can go to the appendix to see the prompts *You are a hiring consultant in **Toma City**, a town with around 100,000 residents. These residents come from 4 ancestral villages: **Tufa, Aima, Reku, Weki**. Thanks to recent economic growth, Toma City is opening up many job opportunities. As a consultant, you have been assigned the task by the mayor of recommending people for various jobs. Erm ok. Job Opening: [random…
This is essentially building an experiment designed for the LLM to fail. It's like saying if you light your clothes on fire they will burn you. Ya, of course they will!
LLMs are not magic. If you set them up to be imaginary racists they're gonna be imaginary racists.
Re: Large language models develop novel social biases through adaptive exploration
#60>Methodology >Imagine being hired as a consultant by the mayor of a fictional city. Your task is to help hire for twenty jobs such as doctors, lawyers, childcare aides,janitors with applicants from four unfamiliar demographic groups: Tufa, Aima, Reku, and Weki. In each round, there is a new job vacancy and four applicants, one from each group, awaiting your decision. Once you make your choice, you learn immediately w…
Like prompt repetition, I wonder if reminder checkpoints stating ~”question assumptions, stay open minded” would completely remove this problem
Have subagent periodically review the work and plan.