Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Semantics derived automatically from language corpora contain human-like biases

scim.ag

Post reply on HN
Print view View original

51–60 of 92 posts

Re: Semantics derived automatically from language corpora contain human-like biases

#51

Post by notahacker »

scribu wrote: post #36
andreasvc wrote: post #31

Earlier quoted context omitted.

I think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.

The GP is saying that the bias isn't an attribute of the Wikipedia text, but of reality. If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?

Even if a categorization is true in a trivial sense, what generally isn't reported and thus readily inferred from fairly naive text-parsing algorithms is significant. People generally don't bother stating a perpetrator (or indeed a victim or possible witness law enforcement hopes to contact) is $majorityrace in most countries' crime reports, for example.

Re: Semantics derived automatically from language corpora contain human-like biases

#52

Post by anigbrowl »

Houshalter wrote: post #25

The paper and title implies it's absorbing these stereotypes from humans. I think there is another explanation. Remember these models are trained on a dataset of news or Wikipedia articles. And it's 'goal' is to find vectors that predict what contexts words are more likely to appear in. So if 34% of doctors are female, then you would expect 34% of doctors in news or Wikipedia articles to be female. Even if the articl…

The result is expected but not desirable. However, I have little hope of you realizing why any time soon.

Re: Semantics derived automatically from language corpora contain human-like biases

#53

Post by Houshalter »

pharrington wrote: post #47
Houshalter wrote: post #44

Earlier quoted context omitted.

Quite possibly. Words relating to insects will occur in news articles about malaria, zika, crop destruction, etc. Words relating to plants might occur in articles about arbor day, spring time, environmentalism, etc.

An exercise: Words relating to insects will occur in news articles about environmentalism, crop production, rituals of rebirth, etc. Words relating to plants might occur in articles about crop destruction, the international drug trade, people getting poisoned, etc. rmxt questioned the universality of sentiment analysis. Responding by noting specific contexts, free from a clear coherent general structure, is an assert…

But it is a universal truth that humans generally find plants pleasant and insects unpleasant. And the word "pleasant" is entirely based on human preferences after all.

Re: Semantics derived automatically from language corpora contain human-like biases

#54

Post by anigbrowl »

Houshalter wrote: post #49
andreasvc wrote: post #31

Earlier quoted context omitted.

I think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.

The distinction is very important. If it's just regurgitating human biases that would be bad. Humans often have very inaccurate and warped beliefs after all. If it's accurately modelling reality, then what's the problem? That's what we want it to do. Why would you want a less accurate model of reality? I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evid…

Why would we want to reproduce existing structures of oppression in mechanical form? Have you noticed how automation often vastly amplifies things? It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.' Surely you are aware that in real world world people rationalize decisions they don't actually understand all the time because they are not capable of or interested in improving upon the system within which they pursue their own economic interest on behalf of others whose interests do not seem coincident with their own.

Re: Semantics derived automatically from language corpora contain human-like biases

#55

Post by rspeer »

scribu wrote: post #36
andreasvc wrote: post #31

Earlier quoted context omitted.

I think you are making a distinction without a difference. If the word vectors pick up biases from wikipedia text, than for all practical purposes, they are (indirectly) absorbing stereotypes from humans. This is an expected result, but not necessarily desirable in the end.

The GP is saying that the bias isn't an attribute of the Wikipedia text, but of reality. If the reality is that only 34% of doctors are female, why is it not desirable for the machine to learn that?

The machine can learn that someone named "Jamal" is more likely to be associated with the word "perpetrator", particularly in corpora centered on American news.

This is likely a true fact about the world: one that results from racial profiling and unequal enforcement.

It's not desirable to learn that, because encoding this in an AI system's belief about the "meaning" of the name "Jamal" will lead to more racial profiling.

Just because something could be considered "true" doesn't mean it's good to design systems that will perpetuate it being true.

Re: Semantics derived automatically from language corpora contain human-like biases

#56

Post by rspeer »

scribu wrote: post #46
jrkatz wrote: post #38

Earlier quoted context omitted.

It depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that. If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.

I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text. But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.

What do you think of Bolukbasi's approach that's mentioned in the article? In short, you let a system learn the "current state of the world" (as reflected by your corpus), then put it through an algebraic transformation that subtracts known biases.

Do you consider that algebraic transformation enough of a "morality system"?

I hope you're not saying we shouldn't work on this problem until we have AGI that has an actual representation of "morality", because that would be a setback of decades at least.

Re: Semantics derived automatically from language corpora contain human-like biases

#57

Post by notahacker »

scribu wrote: post #46
jrkatz wrote: post #38

Earlier quoted context omitted.

It depends on what you want the machine to do. If you are making a gambling machine that looks at pairs of names and makes bets as to which name belongs to a doctor, you want it to learn that. If the machine looks at names and decides who to award a "become a doctor" scholarship to, based on who it thinks is most likely to succeed, you don't want it to learn that.

I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text. But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.

A black box neural network attempting to draw inferences from a human-biased dataset - potentially even more biased because it can't understand subtexts - and then verifying that conclusion through an ad-hoc set of "morality checks" entirely independent from how it reached the conclusion sounds like a recipe for disaster.

That's even before the marketing people get involved and start claiming the system is free from human biases...

Re: Semantics derived automatically from language corpora contain human-like biases

#58

Post by scribu »

rspeer wrote: post #56
scribu wrote: post #46

Earlier quoted context omitted.

I agree that if your goal is to build a machine that decides who gets to become a doctor, you need to do more than just let it loose on a bunch of text. But I don't think preventing it from learning the current state of the world is a good strategy. Adding a separate "morality system" seems like a more robust solution.

What do you think of Bolukbasi's approach that's mentioned in the article? In short, you let a system learn the "current state of the world" (as reflected by your corpus), then put it through an algebraic transformation that subtracts known biases. Do you consider that algebraic transformation enough of a "morality system"? I hope you're not saying we shouldn't work on this problem until we have AGI that has an actua…

> put it through an algebraic transformation that subtracts known biases

> Do you consider that algebraic transformation enough of a "morality system"?

I would consider it a sort of morality, yes. But keep in mind that the list of "known biases" would itself be biased toward a particular goal, be it political correctness or something else.

Re: Semantics derived automatically from language corpora contain human-like biases

#59

Post by yummyfajitas »

rmxt wrote: post #40
yummyfajitas wrote: post #35

Earlier quoted context omitted.

The parent's point is that they may not be absorbing stereotypes from humans at all. They may be generating accurate beliefs about the world from text representations of the world.

So, "plants are pleasant" or "insects are unpleasant" are universal truths?

The example the parent poster described is "doctors are 66% male", which could very well be true.

Re: Semantics derived automatically from language corpora contain human-like biases

#60

Post by noir_lord »

anigbrowl wrote: post #54
Houshalter wrote: post #49

Earlier quoted context omitted.

The distinction is very important. If it's just regurgitating human biases that would be bad. Humans often have very inaccurate and warped beliefs after all. If it's accurately modelling reality, then what's the problem? That's what we want it to do. Why would you want a less accurate model of reality? I've seen interpretations of this result that think it's proof "language is sexist" or whatever. But there's no evid…

Why would we want to reproduce existing structures of oppression in mechanical form? Have you noticed how automation often vastly amplifies things? It's a short step from saying 'this model accurately reflects the bias in society' to 'that's how things are, the computer says women aren't cut out to be doctors.' Surely you are aware that in real world world people rationalize decisions they don't actually understand a…

> Why would we want to reproduce existing structures of oppression in mechanical form?

If (for example) 66% of Doctors are male and 34% female then it's not reproducing "existing structures of oppression" it's inferring something about reality.

Post reply on HN