Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

131–140 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#131

Post by darawk »

jkingsbery wrote: post #52

(Disclaimer: I am an Amazon employee sharing his own experience, but do not speak in any official capacity for Amazon. I don't know anything about the system mentioned in this article.) I am a frequent interviewer for engineering roles at Amazon. As part of the interview training and other forums, we often discuss the importance of removing bias, looking out for unconscious bias, and so on. The recruiters I know at A…

It seems like they did think it through, though? And that's why it's being shelved. I don't really see what the story is here. It seems like the whole process worked exactly as it should - Amazon tried something, it had some unintended consequences, they caught it, and shelved it.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#133

Post by darawk »

gfodor wrote: post #12

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class? If we'r…

All of that is true, but I think the most important question is: compared to what? ML is substantially more transparent than human decision makers. Human decision makers will actively lie to you. ML is a major step forward in correcting these sorts of biases, by making interpretable (relative to humans) models in the first place.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#134

Post by kentm »

electrograv wrote: post #92
kentm wrote: post #67

Earlier quoted context omitted.

Unfortunately using open source work, even if only for programming, introduces all sorts of biases as well. A lot of very competent programmers work at jobs that do not have open source contributions and also have families which limit the time they can spend coding after work.

Good point! I didn’t mean to propose it as a solution, but rather as an enumeration or one possibility I’ve considered, and ruled out because it doesn’t work for everyone.

I didn't mean to assume you did, sorry. Its just fun to be the jerk that points out that hiring without bias is actually really hard.

We've had quite a lot of discussions about this internally, and even with humans at the helm with best of intentions about being unbiased, its really easy for a lot of bias to slip in. Even things like the phrasing of questions can introduce bias (i.e. the ol' apocryphal SAT word association problem that had 'regatta:boat').

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#135

Post by [deleted] »

happytoexplain wrote: post #130
macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

I agree with you, but I have to point out, because it's so common: The "this will be unpopular, but I don't care" preface is, I feel, about as damaging to the perception of whatever you're about to say as "I'm not racist, but". To make an effective point, I think you should avoid, as the first thing you say, painting yourself as an underdog brave enough to speak out by preemptively criticising your audience's reactio…

[deleted]

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#136

Post by arandr0x »

The idea that Amazon is trying to enforce diversity by using an algorithm that is made to detect, and match, patterns boggles the mind. Why, yes, if your recruiting cost function is "is that person just like all the others we hired", you will end-up with a non-diverse workforce, no matter whether the model optimized with this function has 20 layers, 250 hyperparameters, or two legs, two arms and a fast-receding hairline.

You can de-bias by explicitly controlling for gender, but now everyone in your company went to CMU and likes dogs.

The more I see news about what recruiting for ultra-large corporations, the more I think one of two things is true:

* ultra-large corporations are doomed to hire less and less well in a way that is more and more biaised, and we should regulate against such corporations in a way that forces them to redistribute their wealth to SMBs;

* ultra-large corporations need to start exclusively growing through acquisitions, which will have the effect of redistributing their wealth to SMBs, and also of hiring a more diverse base of employees because there is a priori a greater diversity of backgrounds leading to success in the free market than the diversity of backgrounds leading to success in the Amazon interview.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#137

Post by SolaceQuantum »

21 wrote: post #66
komali2 wrote: post #27

Earlier quoted context omitted.

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

> There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models) This one is a bit weird, computer guys were always "nerds" and "geeks" to stay away from.

The rhetorical context here is that human children look for other human adults that they could potentially grow into in order to aim their own dreams and hopes for their adulthood. If a young human boy sees an adult human man pursuing computers, the young human boy learns that being interested in computers is a socially viable construct and this will affect how he pursues his interests in the future. In consequence, if a young human girl does not see any adult human women in computers, she may not understand that that option is available to her and this will affect how she pursues her interests. Although there is some fuzziness in determining this (some children grow up to be trailblazers, others pursue passions regardless of examples).

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#138

Post by darawk »

jakelazaroff wrote: post #53
gfodor wrote: post #12

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class? If we'r…

> What about cases where the features are just merely correlated with a subpopulation of a protected class? This question can be rephrased as "is there a difference between de facto and de jure discrimination?" My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only…

Ok, so if you're hiring professional arm wrestlers, and your model looks at bicep muscle mass, is that discrimination because it selects against women?

If you're hiring therapists, and your candidates take a personality test, and your ML model weights the 'nurturing' feature highly, is that discrimination because it selects against men?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#139

Post by thoughtexplorer »

Anyone who has worked at tech companies and has been involved in the hiring process knows the following:

The best thing to be right now is a woman engineer. You can easily get hired within the week.

Unfortunately this doesn't seem to be well known outside of those involved in hiring.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#140

Post by darawk »

There's something I don't understand in stories like this. It ought to be relatively straightforward to correct biases like this. All you need to do is train a model to explicitly classify gender from resume's, and then use that model to de-gender resume's before passing them on to the hiring model. Is there some reason people aren't doing this?
Post reply on HN