Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#411

Post by Bartweiss » Thu, Oct 11, 2018, 3:56 PM UTC

grogenaut wrote: post #284

Bartweiss wrote: post #171

Earlier quoted context omitted.

> the features are just merely correlated with a subpopulation of a protected class The article notes that Amazon's system rated down grads from two all-women's schools. But it immediately occurs to me to wonder what the algorithm did with candidates from heavily gender-imbalanced schools, which could be much harder to spot. RPI's Computer Science department is about 85% male, while CMU's is just over 50% male. CMU's…

A subtle point you may have missed, amazon knew about and accounted for the gender bias, the scrapped it because of all of the biases that they couldn't identify and were leery of. Most of your suggestions seem to be solving for the known biases, which I believe they did. Also knowing some people who worked on this, they were VERY cognizant of re-encoding biases from the start of the project, it was one of the main r…

I did not at all get from the article "amazon knew about and accounted for the gender bias".

"Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory." I read that as a very different statement - as written, Amazon corrected two specific instances of keyword gender bias by hand, but couldn't reliably prevent further bias (including gender bias) from arising. That's where tricks like "ask the system to classify gender, and then un-train via that data" come in.

(I don't mean you're wrong, just that if gender bias was accounted for more generally, the article should have said so.)

That said, I think our disagreement might just be a miscommunication on what went wrong in the first place. If you know some people involved, maybe you can help clarify the situation?

The article totally fails to explain why "most engineering resumes are from men" led to an algorithm that downrated female resumes. "Most applicants had brown hair" does not produce a system that downrates blondes if you tell it hair color. So the question is - was the training data biased against female applicants (in which case why wasn't it caught before specific outputs needed modification?), or did something else altogether cause this issue (in which case what?)

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#412

Post by notSoRandom27 » Thu, Oct 11, 2018, 4:08 PM UTC

beat wrote: post #206

Dirlewanger wrote: post #202

Earlier quoted context omitted.

It's taking a little longer than it should, but people are finally starting to realize that the actual reason there aren't as many women in software is because they've chosen not to be. They're wired differently and therefore have different interests.

But there are fewer women in software now than there were 30 years ago. Are women today "wired differently" from their mothers? Single-generation changes in behavior aren't genetic. They're social.

> But there are fewer women in software now than there were 30 years ago. Are women today "wired differently" from their mothers?

But are they really ?, do you have any data/reference to back it up, I was under the assumption that there are more people, both men and women, working as programmers, than 30 years ago.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#413

Post by theptip » Thu, Oct 11, 2018, 4:11 PM UTC

tomp wrote: post #190

theptip wrote: post #125

Earlier quoted context omitted.

> "No bias" means that gender is irrelevant False. If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data -- see this article by Chris Stucchio with some images for clarification: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what... "In statistics, a “bias” is defined as a statistical predictor which makes errors that all have the…

I think the comments I replied to mean bias as in “sexist bias”.

Well, it was clear that _you_ think so.

My point was that you should consider the meaning of the word under which the post you're replying to is correct, especially given that the author was claiming specific domain experience.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#414

Post by theptip » Thu, Oct 11, 2018, 4:16 PM UTC

chiefalchemist wrote: post #368

theptip wrote: post #125

Earlier quoted context omitted.

> "No bias" means that gender is irrelevant False. If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data -- see this article by Chris Stucchio with some images for clarification: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what... "In statistics, a “bias” is defined as a statistical predictor which makes errors that all have the…

Question: So technically, the AI is not bias against women per se, but a set of characteristics / properties, that are more common among women. I'm not trying to split hairs (or argue), as much as further clarify the difference between (the common definition of) human bias and that of statistical bias.

> the AI is not bias against women per se

I think I'd make the claim a bit less strongly -- we don't know if there is statistical bias or non-statistical/"gender bias" in the data; both are possible based on what we know.

However exploring the statistical bias possibility, the simple way this could happen is if the data have properties like:

1. For whatever reason, fewer women than men choose to be software engineers 2. For whatever reason, the women that choose to be software engineers are better at it than men

(Note I'm just using hypotheticals here, I'm not making claims about the truth of these, or whether it's gender bias that they are true/false).

Depending on how you've set up your classifier, you could effectively be asking "does this candidate look like software engineers I've already hired"? If so, under the first case, you'd correctly answer "not much". Or you could easily go the other way and "bias" towards women if you fit your model to the top 1% where women are better than men, in our hypothetical dataset.

This would result in "gender bias" in the results, but there's no statistical bias here, since your algorithm is correctly answering the question you asked. It's probably the wrong question though!

Figuring out if/when you're asking the right question is quite difficult, and as the sibling comment rightly pointed out, sometimes (e.g. insurance pricing) the strictly "correct" result (from a business/financial point of view) ends up being considered discriminatory under the moral lens.

This is why we can't just wash our hands of these problems and let a machine do it; until we're comfortable that machines understand our morality, they will do that part wrong.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#415

Post by softawre » Thu, Oct 11, 2018, 4:21 PM UTC

notSoRandom27 wrote: post #408

N0RMAN wrote: post #199

Earlier quoted context omitted.

Just being female is a qualification at your workplace?

Not the parent, but in my current company it definitely helps to be a women, more referral bonus for referring a diversity candidate (post joining), performance targets for managers related to hiring and promoting diversity candidates. Please note there is no active discrimination against men, but preference in some cases would be to hire a women. Gender is the only criteria for diversity in here.

How is a preference for hiring women not discrimination against men?

Would a preference for hiring men be discrimination against women?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#416

Post by theptip » Thu, Oct 11, 2018, 4:28 PM UTC

zaarn wrote: post #398

chiefalchemist wrote: post #368

Earlier quoted context omitted.

Question: So technically, the AI is not bias against women per se, but a set of characteristics / properties, that are more common among women. I'm not trying to split hairs (or argue), as much as further clarify the difference between (the common definition of) human bias and that of statistical bias.

Correct. Computers are very bad at actually discriminating against people, they will pick up a possible bias in a statistical dataset (ie, uses certain sentence structure and is statistically less likely to get or keep the job). Sometimes computers also pick up on statistical truths that we don't like, ie, you assign a ML to classify how likely someone is to pay back their loan and it picks up on poor people and bad…

> the brain can be simplified as a bayesian filter

I agree with everything else in your post, but just wanted to note that while this is true to some extent, the brain is much less rational than a pure Bayesian inference system; there are a lot of baked in heuristics designed to short-circuit the collection of data that would be required to make high-quality Bayesian inferences.

This is why excessive stereotyping and tribalism are a fundamental human trait; a pure Bayesian system wouldn't jump to conclusions as quickly as humans do, nor would it refuse to change its mind from those hastily-formed opinions.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#417

Post by s73v3r_ » Thu, Oct 11, 2018, 4:29 PM UTC

gsich wrote: post #366

s73v3r_ wrote: post #338

Earlier quoted context omitted.

More than that, though. Graduates of all-women colleges were also caught. If you're using school as a data point, that's extremely hard to sanitize.

Then what is the purpose of this? At some point you want this thing to "discriminate" (or "select", if this is a better word) people based on what they have done in life. Which is not negative per se.

But you don't want it to select based on gender.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#418

Post by ArchTypical » Thu, Oct 11, 2018, 5:25 PM UTC

GuB-42 wrote: post #372

dheera wrote: post #324

Earlier quoted context omitted.

The term "AI" is over-hyped. What we have now is advanced pattern recognition, not intelligence. Pattern recognition will learn any biases in your training data. An intelligent enough* being does much more than pattern recognition -- intelligent beings have concepts of ethics, social responsibility, value systems, dreams, ideals, and is able to know what to look for and what to ignore in the process of learning. A du…

We are pattern recognition machines. If you consider pattern matching unintelligent, then machines are more intelligent that we are since they rely more on logic than pattern matching. For the black man = gorilla problem, an untaught human, a small child for instance, can easily make the same mistake. Especially if he has seen few black people. And well educated adults can also make the mistake initially, even if the…

> We are pattern recognition machines.

That's just one function. That's not the entirety of what the brain (and body) does.

> If you consider pattern matching unintelligent,

What do you think pattern matching IS? Round ball round hole does not require intelligence. It requires physics. The convoluted rube goldberg meat machine what we use to do it, doesn't change what it is. Making the choice of will and approximations, are more signs of intelligence, imo.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#419

Post by tomp » Thu, Oct 11, 2018, 5:32 PM UTC

theptip wrote: post #413

tomp wrote: post #190

Earlier quoted context omitted.

I think the comments I replied to mean bias as in “sexist bias”.

Well, it was clear that _you_ think so. My point was that you should consider the meaning of the word under which the post you're replying to is correct, especially given that the author was claiming specific domain experience.

The original was:

> The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

your post said:

> If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data

So I think my interpretation is correct, even though it's not "the technically statistically correct usage". You were referring to the bias of the algorithm (i.e. the mean divergence from the mean in the data), whereas we were referring to the "hiring bias" evident in the data. In fact, your "bias" was mentioned as "lower rankings for women" - i.e. "the algorithm would have (statistical) bias even without (sexist) bias in the data" and I was replying that I think that's false.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#420

Post by mrow84 » Thu, Oct 11, 2018, 8:44 PM UTC

ramblerman wrote: post #404

arandr0x wrote: post #247

Earlier quoted context omitted.

I hope you're actually an American who doesn't like the NFL so you can tell me how well the analogy follows. The conversational equivalent using the NFL example would go something like this: "Why are there no Americans at my favorite chess forum?" "Americans like the NFL. They're just more into brute force and camaraderie, especially American men. Chess can't really appeal to them. I mean, back in the Neolithic, a mo…

I reread this three times, and what a contrived way to look at the world. You are the one coming up with all these labels, and trying to project it into other people's arguments. Let me play the same game: I know a black guy who is president. Does that mean all black men are political? What about the one who just wants to play sports. Will we now call him an athletic black man, instead of just a black man... You can…

My summary of the most recent comments in this thread:

Dirlewanger: group X have property Y arandr0x: members of group X without property Y might be offended by "group X have property Y" courir: as a member of group X without property Y I'm not offended by "most members of group X have property Y" arandr0x: ramblerman: does saying a member of group X has property Y mean all members of group X have property Y?

I think you (ramblerman) have logically inverted the main claim, which is why it doesn't seem to make sense. Behaviour in line with arandr0x' comment seems perfectly reasonable to me - few people take well to poorly fitting generalisations.