Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

51–60 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#51

Post by 21 »

komali2 wrote: post #15
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

I understand your frustration, but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. Maybe in the 90s, but in the early 10s there was tons of policy around it, bosses were setting the culture, we were doing everything "right." But we were still not hiring that many women, simply because hardly any women ever applied. For chem…

I worked in a tech company which was heavily biased in favor of hiring woman, there were special hiring tracks for woman where the interviews were easier and the interviewers received special training in unconscious biases, and the managers received bonuses for having a closer to 50/50 ratio.

It was still just a trickle, for the same reason you stated - very very few woman apply to tech positions.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#52

Post by jkingsbery »

(Disclaimer: I am an Amazon employee sharing his own experience, but do not speak in any official capacity for Amazon. I don't know anything about the system mentioned in this article.)

I am a frequent interviewer for engineering roles at Amazon. As part of the interview training and other forums, we often discuss the importance of removing bias, looking out for unconscious bias, and so on. The recruiters I know at Amazon all take reaching out to historically under-represented groups seriously.

I don't know anything about the system described in the article (even that we had such a system), but if it was introducing bias I'm glad it's being shelved. Hopefully this article doesn't discourage people from applying to work at Amazon - I've found it a good place to work.

To say something about the AI/ML aspect of the article: I think as engineers our instinct is "Here's some data that's been classified for me, I can use ML/AI on it!" without thinking through all that follows, including doing quality assurance. I think a lot of focus in ML (at least in what I've read) has been on generating models, and not nearly enough focus has been on generating models that interpretable (i.e., give a reason along with a classification).

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#53

Post by jakelazaroff »

gfodor wrote: post #12

This is a direct and clear example of bias which made it easy to flag the ML algorithm. But what about ML algorithms that are inducing benefits to groups in less obvious contexts? What about groups that are not so easily identified as being protected classes by simple, human-understandable model features? What about cases where the features are just merely correlated with a subpopulation of a protected class? If we'r…

> What about cases where the features are just merely correlated with a subpopulation of a protected class?

This question can be rephrased as "is there a difference between de facto and de jure discrimination?"

My answer is no, causality doesn't matter here: if feature A is a good predictor that some person belongs in group B and not group C, then filtering out feature As is effectively the same as filtering out only group Bs.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#54

Post by trustmath »

Skrillex wrote: post #24
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

Is this not Amazon trying to not discriminate against women?

Sure. It just sounds like their code and it's results got too complicated to reason about.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#56

Post by fhbdukfrh »

electrograv wrote: post #44

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn…

It doesn't work for the vast majority of professional work because it's not open source and not done in the public

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#57

Post by izzydata »

It is no secret that more men pursue careers in this field. How can you expect any algorithm to produce an equal number of men and woman applicants? If it did this then that would be actually biased in favor of woman.

If this is what they want then they have to feed their neutral algorithm pre-biased data to get their expected results.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#58

Post by kaitai »

kareemsabri wrote: post #45
fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

The whole aim of the AI was to make decisions like the recruiters did -- that is explicitly what they were aiming to do. It might be worth reading the article as it addresses your two ideas (the aim of the project and the fact that the training set was indeed heavily male).

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#59

Post by michaelt »

sanxiyn wrote: post #38
michaelt wrote: post #29

Earlier quoted context omitted.

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Maybe - but maybe not. Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates. If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you p…

Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".

Because you were trying to build a system that would perform CV filtering for your entire company, and you figured Deep Learning would just kinda take care of everything.

You're right you'd want to look at applicants' CVs - I skipped over that to make the numbers readily comprehensible.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#60

Post by fx32s »

strict9 wrote: post #8

At best ai amplifies existing patterns and biases when handling repetitive work. Over and over we hear how Facebook, Twitter, Google, and others will solve the problem of problematic content and bad actors through ai and neural networks. It's a fraud and the digital potemkin village of our era.

AI learns from the training data it's given and copies any biases this data exhibits. Pretty much all software today uses ML in some form to improve their services. I feel it's here to stay and not bad by default. We just have to make sure we are aware of its current limitations.

Facebook is already auto-flagging content this way but it's just a very hard problem (even for humans).

Post reply on HN