Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

111–120 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#111

Post by arayh »

electrograv wrote: post #44

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn…

I agree that resumes are a poor means to distinguish between good and bad candidates. Humans already struggle with the screening process. There's no way an AI can reveal some kind of hidden secret sauce written into all great candidate resumes. This project was doomed to fail from the beginning, in my opinion.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#112

Post by daenz »

LinuxBender wrote: post #98
TangoTrotFox wrote: post #90

Earlier quoted context omitted.

The AI did not have access to gender. It was just word weighting and it turned out that words that could be linked to females ended up with applicants that had a negative outcome. Like the article says the AI ended up giving a negative weight to any resume containing the word women's as in women's [---] club , or those that mentioned certain all women's colleges.

It should be fairly easy to filter / replace all of that. The same logic can apply. Can we add some simple filters and re-train it?

>Amazon edited the programs to make them neutral to these particular terms. But that was no guarantee that the machines would not devise other ways of sorting candidates that could prove discriminatory, the people said.

If you've ever trained a NN, you'll know that they are exceedingly clever in finding patterns that fit what you're training for. You can remove the word "women's" and other obvious things from being considered, but I promise you, if there's another non-obvious patterns that are more likely to apply to the women candidates, the AI will find them and use them.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#113

Post by rfeather »

kaitai wrote: post #78

I'm doing a bunch of ML on a very different data set -- looking at what people eat (survey data). What's interesting to me is that if you do principal component analysis, for instance, there are some differences between the boys & girls in the sample, but they're not very distinct. If you do clustering or random forests on the dietary intakes of the whole cohort, you get mushy and unclear signals. If you split the su…

Off topic, but I'm looking at similar data. Are you talking about a public source (eg NHANES) or something else?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#114

Post by komali2 »

thoughtexplorer wrote: post #49
trustmath wrote: post #2

I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.

> if people would just let them. People do let them. You can't force what people are interested in and you can't let in that which does not exist. In fact many places in tech give preference to women applicants, because they don't apply often and the companies want more women. They're just rare to see. :( There's no grand conspiracy. The truth is much less exciting: Women and men have different preferences, generally…

There's no actual evidence to suggest preference for career type is inherent to being a woman or man. There's plenty showing that women right now have different interests of men, but the cause can easily be societal conditioning - i.e., something that can be fixed.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#115

Post by kareemsabri »

erikpukinskis wrote: post #106
kareemsabri wrote: post #79

Earlier quoted context omitted.

Hey. I did read the article. It doesn’t support the conclusion OP is drawing. The aim of the AI is to “mechanize the search for talent”. It doesn’t care to, nor have any means to, make decisions “like the recruiters did”. Obviously machines don’t make decisions like humans do. They’re trying to reverse engineer an alternate decisions making process from the previous outcomes.

Aren't the "previous outcomes" past hiring decisions though?

Yes, but you have to know what pool you started with. As an overly simplistic example, if a bank used historical mortgage approval records from primarily German neighbourhoods to train AI, it might become racist against non-Germans despite that it’s just an artifact of the demographics of the time. I think it just shows how not ready for prime time AI is.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#116

Post by linuxftw »

mcherm wrote: post #28
ummonk wrote: post #4

Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.

Why not? Seriously: let us take as given that the AI models are biased. Will you also admit that the existing processes are biased? If so, then what we need to ask is which is MORE biased. It might be complaining that we shouldn't release self-driving cars because on rare occasions they cause accidents. There is, however, another criterion besides how biased it is: how biased it will be in the future. Human-driven pr…

Machines are not intuitive or nuanced. They are incapable of learning and formulating abstract though, only identifying patterns and optimizing for some desired outcome.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#117

Post by HashHishBang »

Hold on here. This article seems to have buried a pretty important piece of information wayyy down in the middle of the text.

> Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.

Granted an article isn't going to get as much attention without an attractive headline but that seems a far more likely reason to have an AI based recruiting recommendation scrapped. The discovery of a negative weight associated with "women's" or graduates of two unnamed women's colleges is notable but if it's tossing out results "almost at random" then...well there seems to be bigger problems?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#118

Post by komali2 »

21 wrote: post #66
komali2 wrote: post #27

Earlier quoted context omitted.

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

> There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models) This one is a bit weird, computer guys were always "nerds" and "geeks" to stay away from.

Wouldn't being an outcast make you even more attracted to heroes of your "outcast class?" Because, presumably, the hero had to overcome so much more for society to recognize them.

Depending on the era, we had Einstein, Turing, feinman. Kids my age had Gates (literally the richest man on the planet for my entire formative years), Jobs, Bill Nye. Little further along are the myth busters crew, musk...

We have plenty of heroes to pick from :)

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#119

Post by trustmath »

komali2 wrote: post #108
arbitrage wrote: post #46

Earlier quoted context omitted.

> but in my experience recruiting, the primary reason behind there being less women getting hired into engineering roles is almost never raw sexism. In my experience in the industry, this is a laughable statement to make. It's a shame that unless one is a victim of unconscious, systemic bias, one is so much less likely to acknowledge it as a problem, that actually exists, and hurts people all the time.

I don't understand what you're saying - is your argument that only victims of unconscious bias will recognize it as a problem? If that's your point, I guess my counter argument is myself, not a victim, very aware of the problem, and acknowledging it as a problem as my post. There's also the victims of conscious bias that would probably be able to acknowledge the problem... Am I misreading your post?

I don't believe in unconscious thought, but if unconscious thought were real, it would be obvious that the only people who can be aware of it's effects are the people who can identify a difference between those two states of being, and be able to reduce that down to an model/abstraction/statement.

It doesn't matter if it's intentional or not to a victim. It's still the same system, same cause and effect, same yield of powerlessness.

Whichever way you want to see it.

+ If it's intentional it's not unconscious.

+ If it's unconscious it's part of a culture that tolerates the behavior to the point that it doesn't get questioned.

+ If it does get questioned, eventually people are just playing dumb or it becomes intentional - if it's provable that it continues to occur.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#120

Post by r00fus »

kareemsabri wrote: post #79
kaitai wrote: post #58

Earlier quoted context omitted.

The whole aim of the AI was to make decisions like the recruiters did -- that is explicitly what they were aiming to do. It might be worth reading the article as it addresses your two ideas (the aim of the project and the fact that the training set was indeed heavily male).

Hey. I did read the article. It doesn’t support the conclusion OP is drawing. The aim of the AI is to “mechanize the search for talent”. It doesn’t care to, nor have any means to, make decisions “like the recruiters did”. Obviously machines don’t make decisions like humans do. They’re trying to reverse engineer an alternate decisions making process from the previous outcomes.

So the recruiters may or may not have been biased, but if the previous outcomes were (based on the candidate pool) then the AI is sure to have been "taught" that bias.

Unless Amazon is willing to accept a) another pool of data or b) that the data will yield bias and apply a correction, the AI is almost guaranteed to be taught the bias.

Post reply on HN