Amazon scraps secret AI recruiting tool that showed bias against women
1–10 of 433 posts
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#2I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#3I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got).
Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#4Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#5I'm reminded of why Watson failed, and the problem with ml and ai in general- you can't peek under the hood to see why something happened, or how to keep it from happening without a lot of time, a lot of hard work, and a whole lot of carefully groomed data.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#6I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
Your comment will probably end up buried, but it does raise the question - if they want more female employees, was the issue in the training data, or their recruitment process?
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#7I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon.
Without significant oversight and manual tuning, any training dataset based wholly or in part on current employees is going to demonstrate a bias against women, because there are strictly fewer women. Moreover it's likely that (for a variety of reasons, both intrinsic and extrinsic) fewer women succeed in the interview process as a ratio compared to the number who actually apply.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#8At best ai amplifies existing patterns and biases when handling repetitive work. Over and over we hear how Facebook, Twitter, Google, and others will solve the problem of problematic content and bad actors through ai and neural networks. It's a fraud and the digital potemkin village of our era.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#9I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
That gives rise to a very interesting concept: ML-based bias assessments. If you take some real-life hiring data (or other applications such as sentencing or generally human behavior data) and train the AI on it, then run it through a bunch of tests to see whether there's bias, that can reveal trends in the underlying training data.
I can't imagine this not already being a thing, but I haven't really heard of people using this method.
Re: Amazon scraps secret AI recruiting tool that showed bias against women
#10I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
Maybe, but it seems more likely that the model just didn't work.
From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"