Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#161

Post by petsormeat » Wed, Oct 10, 2018, 4:34 PM UTC

briandear wrote: post #101

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

That is very interesting. What are the stats for unemployed female engineers? Is there a shortage of female representation because female engineers aren’t being hired or is there a shortage because there are actually fewer female engineers? There is a shortage of male therapists and kindergarten teachers: is that because males aren’t being hired or because there are fewer of them in existence?

A lot of us drop out mid-career. I'd argue there's more a shortage of female engineering directors, VPs, CTOs.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#162

Post by [deleted] » Wed, Oct 10, 2018, 4:34 PM UTC

briandear wrote: post #101

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

That is very interesting. What are the stats for unemployed female engineers? Is there a shortage of female representation because female engineers aren’t being hired or is there a shortage because there are actually fewer female engineers? There is a shortage of male therapists and kindergarten teachers: is that because males aren’t being hired or because there are fewer of them in existence?

[deleted]

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#163

Post by kareemsabri » Wed, Oct 10, 2018, 4:35 PM UTC

zby wrote: post #151

kareemsabri wrote: post #45

Earlier quoted context omitted.

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

"they could be unbiased while still amassing a data set that skewed heavily male" - this sounds like a self contradiction

Is the NBA biased against white guys?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#164

Post by gfodor » Wed, Oct 10, 2018, 4:35 PM UTC

darawk wrote: post #138

jakelazaroff wrote: post #53

Earlier quoted context omitted.

> What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only…

Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women? If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?

Separate it one step further -- hiring decisions are all too easy to pin as discriminatory. What if a site shows ads for a special offer on protein shakes for people with higher bicep mass since they are in the target market? Is that discriminating against women?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#165

Post by jakelazaroff » Wed, Oct 10, 2018, 4:36 PM UTC

darawk wrote: post #138

jakelazaroff wrote: post #53

Earlier quoted context omitted.

> What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only…

Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women? If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?

Underlying your examples is the implication that a preference shouldn't be considered discriminatory if the trait being selected for correlates with fitness. I agree with this position!

What I don't agree with is the assumption that, in this case, the preferred traits do correlate with fitness, since there's at least one — gender — for which this model is biased even though it has no apparent correlation.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#166

Post by electrograv » Wed, Oct 10, 2018, 4:37 PM UTC

stef25 wrote: post #103

electrograv wrote: post #44

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn…

> author-blind reviews Should I ever be in a position to hire a colleague, I wouldn't ever do so without having a chat with them. I spend 8hrs a day in an office with my colleagues (sometimes more than with my wife & kid) and the ones I can't stand is about the only thing wrong with my job. If we can't even see the person's face over some gender bias hysteria then I wonder how the hell we got here. People should just…

I’m not saying that all candidates should be interviewed without ever meeting or interviewing the candidate face-to-face; rather, I was speaking specifically of the coding interview portions. In these segments, face-to-face doesn’t really matter IMO, since it’s all about the candidate’s problem-solving ability.

Yes there’s a lot of “bias hysteria” out there, as you put it, but I would dispute that advocating “author-blind meritocracy” falls into that category.

Quite the contrary: An author-blind review process would actually make any bias impossible — either for or against any particular identity group. It seems to me most people should be able to get behind that, but maybe I’m wrong.

In fact, the main opposition to author-blind meritocracy is the “post-meritocracy” movement which is slowly making its way into open-source projects codes of conduct.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#167

Post by amanaplanacanal » Wed, Oct 10, 2018, 4:38 PM UTC

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Two comments:

1. The issue is certainly bigger than hiring. In the many years between birth and looking for a job, there are a lot of societal pressures that will impact what eventual careers people end up in.

2. Hiring managers are people. They are not perfect. They have biases. If someone expects an engineer to look, talk, and act a certain way, that can impact their decision making completely independent of the fact that they want to hire the best people for their company.

Bonus third point: I still see a whole lot of "We want to make sure that the hire fits on the team." This is completely natural, and comes with its own set of built-in biases.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#168

Post by guitarbill » Wed, Oct 10, 2018, 4:38 PM UTC

dudul wrote: post #149

guitarbill wrote: post #143

Earlier quoted context omitted.

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ". And th…

I don't understand how your example is an evidence that data came from mean who didn't want to hire women. These interview processes exist sure, and I personally find them idiotic, but is there evidence that they disqualify women more than men?

From this article/Amazon - no. From my personal experience, yes.

After we moved to a logic-based test, we were able to hire several more women from interesting disciplines including psychology, math, and biology. The tests involved technical problems written in a general way. For example, thread scheduling was written to instead involving painters, rooms, and drying time. We were able to hire 4 women on a ~50 person team in a very short time, and it worked out pretty well.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#169

Post by briandear » Wed, Oct 10, 2018, 4:38 PM UTC

izzydata wrote: post #57

It is no secret that more men pursue careers in this field. How can you expect any algorithm to produce an equal number of men and woman applicants? If it did this then that would be actually biased in favor of woman. If this is what they want then they have to feed their neutral algorithm pre-biased data to get their expected results.

I think it isn’t equality in numbers but in activity screening out women.

So there may still be a 8:1 ratio of men:women but according to the article, it would seem than even that 1 women would have been negatively impacted.

I am not arguing the validity of their conclusions, just saying that I don’t think that it’s about equal numbers: given a male and female applicant, according to the article, women would have had an unequal chance of passing the screening. (Now if that inequality was for valid reasons, that to me, is an open question, but the article indicates that there was unjustifiable bias.)

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#170

Post by kareemsabri » Wed, Oct 10, 2018, 4:38 PM UTC

BurningFrog wrote: post #123

kareemsabri wrote: post #45

Earlier quoted context omitted.

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

Control question for if you're making a certain intellectual mistake. The data set will also have skewed heavily against people named "David". Probably only ~1% of the successful applicants. Would you also expect the machine to be biased against candidates named David?

I would expect the AI isn't fed names as an input, but rather things Amazon wants to weigh like experience, awards and education.