Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

41–50 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#41

Post by kaitai »

exo762 wrote: post #11
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

Why not "women are just not as interested in math, logic and computer science to pursue it AS OFTEN as men"? Why are you not considering this possibility?

If these women are applying for tech jobs at Amazon, they're by definition interested. The uninterested (male or female) are not relevant to this discussion.

I must say, I am frustrated by this being brought up in every discussion of women & STEM. Want to discuss the leaky pipeline from physics PhDs to full professor in physics? "Maybe those women did a PhD in physics despite not being interested, and they just didn't notice before!"

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#42

Post by patwolf »

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed.

I see four possibilities here:

1. The algorithm was designed in a completely inept fashion

2. The algorithm design was sound, but ultimately ineffective

3. The algorithm was sound and effective, but results were considered discriminatory.

4. There's something biased about how employees are rated--the data that would feed into the algorithm, which is possibly more of a human element.

Edit: Added fourth possibility

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#43

Post by mcguire »

throwawaymath wrote: post #7
ncallaway wrote: post #3

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based…

"...any training dataset based wholly or in part on current employees is going to demonstrate a bias against women,..."

"Demonstrate" may be the wrong word. How about "encode". And in the future, "enforce".

[Edit] On second thought, it would seem like there would be a way to filter out a "raw proportions bias" (like 80% of the resumes in the data set are male) before training.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#44

Post by electrograv »

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa).

I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn’t work for every kind of engineering/science (edit: and also would introduce bias against professionals too busy for open source.)

Regarding bias — it seems the only way to truly eliminate it (including unconscious bias) is author-blind reviews, i.e. reviewing code written by a candidate without knowing anything about that candidate’s identity. (And the nice thing about code is it usually doesn’t signal any identity traits of the author via side channels.)

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#45

Post by kareemsabri »

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that female resumes don’t match, as closely, the resumes of successful past candidates. Perhaps I’m ignorant about AI, but I don’t see why the number of candidates of each gender shouldn’t increase the strength of the signal. “Aggressiveness” in the resume may be correlated but not causal. If the AI was fed the heights of the candidates, it might reject women for being too short, but that would not indicate height is a criteria of Amazon recruiters hiring.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#46

Post by arbitrage »

komali2 wrote: post #15
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

I understand your frustration, but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. Maybe in the 90s, but in the early 10s there was tons of policy around it, bosses were setting the culture, we were doing everything "right." But we were still not hiring that many women, simply because hardly any women ever applied. For chem…

> but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism.

In my experience in the industry, this is a laughable statement to make. It's a shame that unless one is a victim of unconscious, systemic bias, one is so much less likely to acknowledge it as a problem, that actually exists, and hurts people all the time.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#48

Post by Novashi »

mcherm wrote: post #28
ummonk wrote: post #4

Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.

Why not? Seriously: let us take as given that the AI models are biased. Will you also admit that the existing processes are biased? If so, then what we need to ask is which is MORE biased. It might be complaining that we shouldn't release self-driving cars because on rare occasions they cause accidents. There is, however, another criterion besides how biased it is: how biased it will be in the future. Human-driven pr…

>Why not?

It has to be better than humans.

That requires the rare genius to fix the problem, or we have to accidentally invent something better than ourselves.

This is a common defense of autopilot systems in cars: it's not perfect, but it's better than the average.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#49

Post by thoughtexplorer »

trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

> if people would just let them.

People do let them. You can't force what people are interested in and you can't let in that which does not exist. In fact many places in tech give preference to women applicants, because they don't apply often and the companies want more women. They're just rare to see. :(

There's no grand conspiracy. The truth is much less exciting: Women and men have different preferences, generally speaking.

Women are more interested in people (i.e. healthcare). Men are more interested in things (i.e. engineering).

Most nurses are women. That's not because women are activity trying to keep men out. It's because fewer men are interested or apply!

We also don’t see women in the most dangerous jobs. No one seems to have a problem with that, just as no one has a problem with most healthcare jobs being dominated by women. As they shouldn't, because people should be allowed to pursue and apply to what they want to.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#50

Post by sanxiyn »

patwolf wrote: post #42

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed. I see four possibilities here: 1. The algorithm was designed in a completely inept fashion 2. The algorithm design was sound, but ultimately ineffective 3. Th…

My current hypothesis is that training data was too noisy.
Post reply on HN