Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

81–90 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#81

Post by tomp »

lalaland1125 wrote: post #40
fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

> The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates). This is incorrect. The key thing to keep in mind is that they are not just predicting who is a good candidate, they are also ranking by the certainty of their prediction. Lower numbers of female candidates could plausibly lead to low…

> The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

I don't think that's true. "No bias" means that gender is irrelevant (i.e. its correlation with outcome is 0%). Therefore the system shouldn't even take it into account - it would evaluate both men and women just by other criteria (experience, technical skills, etc), and it would have equal amounts of data for both (because it wouldn't even see them as different).

You need bias to even separate the dataset into distinct categories.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#83

Post by karrotwaltz »

patwolf wrote: post #42

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed. I see four possibilities here: 1. The algorithm was designed in a completely inept fashion 2. The algorithm design was sound, but ultimately ineffective 3. Th…

From the article:

> Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

It looks like the bias wasn't the only flaw.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#84

Post by fuscy »

kareemsabri wrote: post #45
fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

I'm going to make a supposition here but one of the first things I think they did (especially when trying to fix the AI) was to balance and normalize the data so that there would be no skew between men and women number of records in the data set.

If my supposition is correct then the other parameters are at fault here from which gender and language used stick out.

Another supposition I'm going to make is that they even removed the gender from the data set so that AI didn't know it, but cross-referencing still showed "faulty" results due to hidden bias that the AI can pick up, like language used.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#85

Post by whatgoodisaroad »

I feel like many comments here are taking this story on face value, but the cynic in me reads this as a planned leak to skapegoat a (hitherto unknown) AI system for their existing hiring biases. Public perception of AI is more aware of model biases nowadays, and we seem all to willing to accept this explanation over the simpler explanation that tech hiring at Amazon is broken in the same way it is everywhere.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#86

Post by trustmath »

thoughtexplorer wrote: post #49
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

> if people would just let them. People do let them. You can't force what people are interested in and you can't let in that which does not exist. In fact many places in tech give preference to women applicants, because they don't apply often and the companies want more women. They're just rare to see. :( There's no grand conspiracy. The truth is much less exciting: Women and men have different preferences, generally…

Please read the article. This article is about automated reasoning that discards resumes that are strongly correlated to resumes of women.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#87

Post by al_chemist »

Alice: Our AI says that graduates from that Boston college are poor job candidates.

Bob: Seems reasonable, that college's teaching quality is bad.

--

Alice: Our AI says that graduates from that all-women college are poor job candidates.

Bob: That can't be right, we all know that women are great job candidates whichever college they graduated.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#88

Post by kipchak »

komali2 wrote: post #27
exo762 wrote: post #11

Earlier quoted context omitted.

Why not "women are just not as interested in math, logic and computer science to pursue it AS OFTEN as men"? Why are you not considering this possibility?

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

Referencing D. Schmitts article referenced in the BBC article, he's quoted as saying

>"that using someone's sex to work out what you think their personality will be like is "like surgically operating with an axe"."

Being phrased by the article as a dismissal of Damore, along with G. Rippon's statements However in the article Schmitt is quoted from, he writes that

>"Culturally universal sex differences in personal values and certain cognitive abilities are a bit larger in size (see here), and sex differences in occupational interests are quite large. It seems likely these culturally universal and biologically-linked sex differences play some role in the gendered hiring patterns of Google employees. For instance, in 2013, 18% of bachelor's degrees in computing were earned by women, and about 20% of Google technological jobs are currently held by women."

He goes on to write that Pyschological sex differences might lead to less than 50% of technology employees being women.

This seems to disagree with Professor Rippon's opinion that

>"but even if you accepted the idea that there are some biological differences, all researchers would assert that they're so tiny that there's no way that they can explain the kind of gender gap that's apparent at Google."

I think there's reason to consider both the societal reasons women might be pressured and excluded from STEM-ey fields, as well as potential inherent differences in interest, and that they can both coexist as considerations, and agree that a biased AI is unhelpful, and many women lack a fair shot of success, however disagree that there is nothing useful in Damore's perspective.

Additionally if such inherent differences are distributed on a bell curve, it would make sense that at cases further along the trail that small differences in populations and their medians are more pronounced.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#89

Post by [deleted] »

fhbdukfrh wrote: post #56
electrograv wrote: post #44

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn…

It doesn't work for the vast majority of professional work because it's not open source and not done in the public

[deleted]

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#90

Post by TangoTrotFox »

LinuxBender wrote: post #64

What would happen if you removed gender from all HR and recruiting systems and then retrain the AI? Or for that matter, remove ethnicity, age, creed, etc... Is there any reason we need to be more specific? Race: Human Gender: Yes Age of legal contractual consent: Yes

The AI did not have access to gender. It was just word weighting and it turned out that words that could be linked to females ended up with applicants that had a negative outcome. Like the article says the AI ended up giving a negative weight to any resume containing the word women's as in women's [---] club, or those that mentioned certain all women's colleges.
Post reply on HN