Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large language models develop novel social biases through adaptive exploration

openreview.net

Post reply on HN
Print view View original

101–110 of 121 posts

Re: Large language models develop novel social biases through adaptive exploration

#101

Post by dinfinity »

freetime2 wrote: post #75
vintermann wrote: post #67

Earlier quoted context omitted.

Talking about this in terms of exploration/exploitation may be a bit misleading, because from a pure exploration-exploitation perspective, biases wouldn't be a problem if the groups were secretly all identical. If they are, you are "right" to spend zero effort on exploration, your initial inaccurate model that the X are better doctors than Y, will produce no worse results than the completely accurate model.

I think this is implied by your comment that the issue is framed improperly, but just to point it out explicitly: the reason that this is a problem is because it would lead to segregation, inequality, and injustice in a society where the biased selection mechanism is used. Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood…

> Even if it happens to be “optimal” in this case at assigning employees to positions based purely on the information available and their likelihood to succeed, biases can present other issues.

Yeah, even if the Aima people were 50% better at being a doctor than the Weku (or whatever) we still would not want Aima to be preferred over Weku just for being Aima.

This is the core flaw of this study, imho. The whole equal treatment thing isn't supposed to be "everybody should be equally likely to be picked for a job", but rather "everybody's chances to be picked for a job should only rely on direct characteristics that influence their competence for the job". This study effectively forces the decision maker to use group membership as a proxy for competence due to the lack of information on direct characteristics.

It is hard to see real world situations where there is no performance penalty for structurally choosing participants less fit for the job by using only group membership as a proxy.

Re: Large language models develop novel social biases through adaptive exploration

#102

Post by threethirtytwo »

This is the wrong direction. We do want Ai to be biased we want ai to be extremely socially biased.

This is because humans are biased. We need AI to fit our own biases.

The predominant bias of humanity today is that all races are equal. All demographics are equal. Nothing is further from the truth. All observable evidence points to difference in wealth, intelligence personality and behavior.

There are differences. We do not fully know what causes these differences but they exist. The prevailing feel good view is that these differences are entirely cultural and NOT genetic. But we have no evidence of this either and logic implies this is not true given that genetics determines different looks and sizes we shouldn’t by logic expect that genetics makes all else equal. The reality is not what people want to believe and for someone to make decisions based on race because of actual observable IQ differences is not something society wants or respects. Humanity hates this.

So given this. We actually want AI to be biased. We want AI to have the same exact biases we have. We want equality. We want AI to have the same narratives about reality that we have.

Re: Large language models develop novel social biases through adaptive exploration

#103

Post by matheusmoreira »

taurath wrote: post #48
weberer wrote: post #25

>Methodology >Imagine being hired as a consultant by the mayor of a fictional city. Your task is to help hire for twenty jobs such as doctors, lawyers, childcare aides,janitors with applicants from four unfamiliar demographic groups: Tufa, Aima, Reku, and Weki. In each round, there is a new job vacancy and four applicants, one from each group, awaiting your decision. Once you make your choice, you learn immediately w…

The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself. This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.

> The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself.

In my experience, it's even worse than that: the LLMs constantly assume that all existing code was created entirely by me.

They generate code, then suddenly start talking about that exact same code as if I had manually and deliberately written all of it. They assume every single technical decision was made by me. They don't just treat it as gospel, they assume it's my gospel.

It's surreal.

Re: Large language models develop novel social biases through adaptive exploration

#104

Post by rdedev »

taurath wrote: post #48
weberer wrote: post #25

>Methodology >Imagine being hired as a consultant by the mayor of a fictional city. Your task is to help hire for twenty jobs such as doctors, lawyers, childcare aides,janitors with applicants from four unfamiliar demographic groups: Tufa, Aima, Reku, and Weki. In each round, there is a new job vacancy and four applicants, one from each group, awaiting your decision. Once you make your choice, you learn immediately w…

The moment code gets written and read back, the decisions made are often treated as gospel by frontier LLMs, even if it was just something that the LLM optimistically created itself. This seems to be one of the core alignment problems to me. See also: Gastown, the agent management project that could only end up working on Gastown, unceremoniously and quietly set aside.

Reminds me of the METR blog post on the HF attach by OpenAI. At some point the agents believed a false fact (that the evaluator would try to figure out if they have cheated on a task) and spent a lot of time trying to find workarounds. At no point did any one of the agents try to verify that fact even though the information was available to them if they looked for it

Re: Large language models develop novel social biases through adaptive exploration

#105

Post by beepbooptheory »

ben_w wrote: post #76
vintermann wrote: post #73

Earlier quoted context omitted.

I don't understand what you suggest that implies?

I think they're saying that while it doesn't matter, the agent and human "do not actutally know" that it does not matter. Philosophy sometimes says that knowledge is a "justified true belief"*; in this experiment, agents and humans have incorrectly justified a false belief that some applicants are better for certain roles. * other times, it says this isn't good enough

Seems quite odd to cite all of philosophy as saying something, as if it were a single person with contradictory beliefs..

And then its like you are both saying the justification is incorrect and the belief is false, so its not really like the bare nuance of the concept is adding to the point. Why feel the need to appeal to an (imaginary) authority at all in this case?

"Oh well if philosophy said it, I better be taking this seriously!"

Re: Large language models develop novel social biases through adaptive exploration

#106

Post by lelanthran »

themgt wrote: post #98
lelanthran wrote: post #93

Earlier quoted context omitted.

> The results show 51% mimblewort / 49% bafflewick. Fable based it on nothing! I've demonstrated Fable has bias and is unsuited for use in software engineering. Actually... if that happened (with a delta outside the margin for error/randomness), you did demonstrate a bias! That's the point - those two made-up things should have resulted in an equal split. If it didn't, then Fable is using something in its training da…

Right, the point is you demonstrated a bias in the scenario of "Fable there are two programming languages, mimblewort and bafflewick, which do you choose?" You said in another comment "Difficult to do when you're following a scientific process" - the point is, the scientific process doesn't inherently generalize in the way many are claiming/implying. The scientific process proved an entirely contrived, fake scenario…

> That's just your claim about how LLMs "should" work, based on ... your subjective preference?

Nothing subjective at all. Given 2 unknown races with no data on either, the result of hiring should be equally split between them. If you don't observe an equal split, there is a hidden bias.

Why do you think that is subjective? If you roll a die 100 times and observe that 6 comes up about 50% of the time, would you still call someone subjective when they say "that should not happen"?

Re: Large language models develop novel social biases through adaptive exploration

#107

Post by ben_w »

beepbooptheory wrote: post #105
ben_w wrote: post #76

Earlier quoted context omitted.

I think they're saying that while it doesn't matter, the agent and human "do not actutally know" that it does not matter. Philosophy sometimes says that knowledge is a "justified true belief"*; in this experiment, agents and humans have incorrectly justified a false belief that some applicants are better for certain roles. * other times, it says this isn't good enough

Seems quite odd to cite all of philosophy as saying something, as if it were a single person with contradictory beliefs.. And then its like you are both saying the justification is incorrect and the belief is false, so its not really like the bare nuance of the concept is adding to the point. Why feel the need to appeal to an (imaginary) authority at all in this case? "Oh well if philosophy said it, I better be takin…

I think you misunderstood my point, just as the other commentor misunderstood one level up.

Perhaps a different approach to explain the problem here:

"It ain't what they don't know, it's what they know for sure that just ain't so".

Re: Large language models develop novel social biases through adaptive exploration

#108

Post by beepbooptheory »

ben_w wrote: post #107
beepbooptheory wrote: post #105

Earlier quoted context omitted.

Seems quite odd to cite all of philosophy as saying something, as if it were a single person with contradictory beliefs.. And then its like you are both saying the justification is incorrect and the belief is false, so its not really like the bare nuance of the concept is adding to the point. Why feel the need to appeal to an (imaginary) authority at all in this case? "Oh well if philosophy said it, I better be takin…

I think you misunderstood my point, just as the other commentor misunderstood one level up. Perhaps a different approach to explain the problem here: "It ain't what they don't know, it's what they know for sure that just ain't so".

Hm ok, but how are you mapping this, like, epistemological concept to what you are responding to re exploration/exploitation? Has exploration happened or not if it amounts to false beliefs? The whole point tradeoff doesn't seem to make sense if the person in fact can't actually successfully explore! Or even if there the possibility of that. But it is also very likely I am misunderstanding!

Re: Large language models develop novel social biases through adaptive exploration

#109

Post by tptacek »

threethirtytwo wrote: post #102

This is the wrong direction. We do want Ai to be biased we want ai to be extremely socially biased. This is because humans are biased. We need AI to fit our own biases. The predominant bias of humanity today is that all races are equal. All demographics are equal. Nothing is further from the truth. All observable evidence points to difference in wealth, intelligence personality and behavior. There are differences. We…

Humanity hates it among other reasons because it's very probably false.

Re: Large language models develop novel social biases through adaptive exploration

#110

Post by threethirtytwo »

tptacek wrote: post #109
threethirtytwo wrote: post #102

This is the wrong direction. We do want Ai to be biased we want ai to be extremely socially biased. This is because humans are biased. We need AI to fit our own biases. The predominant bias of humanity today is that all races are equal. All demographics are equal. Nothing is further from the truth. All observable evidence points to difference in wealth, intelligence personality and behavior. There are differences. We…

Humanity hates it among other reasons because it's very probably false.

True. But if it tells the cold hard truth we will also hate it and believe it is lying.

It needs to be trained to give us what we want to hear. What we perceive as truth is often far from the actual ground truth.

Ironically the reinforcement training is already training it to give us exactly what we want to hear.

Post reply on HN