Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#141

Post by IshKebab » Wed, Oct 10, 2018, 4:19 PM UTC

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

They didn't scrap it because of this gender problem. That wasn't why it failed. They scrapped it because it didn't work anyway.

Note the title is "Amazon scraps secret AI recruiting tool that showed bias against women" not "Amazon scraps secret AI recruiting tool because it showed bias against women". But I guess the real title is less clickbaity - "Amazon scraps secret AI recruiting tool because it didn't work".

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#142

Post by bglusman » Wed, Oct 10, 2018, 4:20 PM UTC

zdragnar wrote: post #5

I'm reminded of why Watson failed, and the problem with ml and ai in general- you can't peek under the hood to see why something happened, or how to keep it from happening without a lot of time, a lot of hard work, and a whole lot of carefully groomed data.

This is especially memorable/well treated in the Cory Doctorow novella/short story "Human Readable"

https://craphound.com/stories/2005/10/12/human-readable/

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#143

Post by guitarbill » Wed, Oct 10, 2018, 4:20 PM UTC

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women?

One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ".

And the number of people who are hired who can barely communicate effectively is still shocking. Very few interview questions focus on communication outside the technical realm.

So you can argue there is a bias in recruiting, simply because different people have different criteria for what the best traits/skills to look for is - even though everybody has the same goal, hiring the "best".

I'd also caution about taking Reuters too seriously though. Seems that they've only focused on the gender issue, but this is the money quote:

> With the technology returning results almost at random, Amazon shut down the project, they said.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#144

Post by dudul » Wed, Oct 10, 2018, 4:20 PM UTC

bicubic wrote: post #124

commandlinefan wrote: post #110

Earlier quoted context omitted.

Yeah - I'm not even expressing an unpopular opinion, just asking a (leading) question: where are all these women who are chomping at the bit to get into _technical_ positions like programming but find themselves being turned away by biased recruiters? I've never even seen somebody _claim_ that they were a woman who couldn't find a tech job, just people wondering where all the women were.

Last place I worked at had to invest in additional resources to hire several females because the office was mostly male and applicants were overwhelmingly male. We did eventually find a few great female applicants, but it took a lot of work and a lot of time dedicated specifically to that goal. There does not exist some magical undiscovered pool of talented female engineers that are being turned away by biased recrui…

Can you share a little more what you did specifically to find female candidates?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#145

Post by moate » Wed, Oct 10, 2018, 4:22 PM UTC

bicubic wrote: post #74

patwolf wrote: post #42

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed. I see four possibilities here: 1. The algorithm was designed in a completely inept fashion 2. The algorithm design was sound, but ultimately ineffective 3. Th…

We're talking about Amazon, one of the biggest powerhouse ML employers. I don't buy that the model was poorly designed or ineffective. They also didn't just scrap the model without understanding how or why it failed to meet its objectives. And whatever the cause was, it was not the poor quality of the training data. They tried to stop the model from downranking women based on obvious keywords, only to find it learnin…

Well it's also what you're training for. If you're building a machine to return "John Doe, The Company's Best AI Dev" then there's a few things that you might get back from a working and effective machine. The problem is that while the machine is doing the best to replicate a John Doe, the humans who designed the machine might realize that there's a lot of variables in what they're looking for that scoping the design is impossibly complex.

Basically, people WANT bias, but they want specific bias. One of the difficulties in training a machine to understand what you find as viable bias vs problematic bias is all the tiny nuances. Yes, you want a great engineer on paper, but you also need to have as diverse a cast as you can in your company (both for optics and creative solutions) AND you need to get people you can afford AND you need someone who's enjoyable to work with etc etc.

Hiring is always going to be part art and part science. There will always be some type of discrimination because of the perceptions of what makes a good qualification for the job. Any hiring group is just going to have their own hierarchy of what they think are the most important skills to have. You can only approach perfection/unbiased hiring, you can never actually achieve it.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#146

Post by crooked-v » Wed, Oct 10, 2018, 4:22 PM UTC

commandlinefan wrote: post #110

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Yeah - I'm not even expressing an unpopular opinion, just asking a (leading) question: where are all these women who are chomping at the bit to get into _technical_ positions like programming but find themselves being turned away by biased recruiters? I've never even seen somebody _claim_ that they were a woman who couldn't find a tech job, just people wondering where all the women were.

They all stopped joining up around 1985. https://www.npr.org/sections/money/2014/10/21/357629765/when...

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#147

Post by SolaceQuantum » Wed, Oct 10, 2018, 4:23 PM UTC

kipchak wrote: post #88

komali2 wrote: post #27

Earlier quoted context omitted.

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

Referencing D. Schmitts article referenced in the BBC article, he's quoted as saying >"that using someone's sex to work out what you think their personality will be like is "like surgically operating with an axe"." Being phrased by the article as a dismissal of Damore, along with G. Rippon's statements However in the article Schmitt is quoted from, he writes that >"Culturally universal sex differences in personal val…

The jump here in the data being displayed is that it is making a correlation between sex differences and bachelor's degree demographics. Very little in that rhetoric actually has logical sense such that we know that we are missing(usually) at least close to two decades of cultural and social conditioning before the bachelors degree. That's plenty of time to systematically condition women against specific fields.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#148

Post by crooked-v » Wed, Oct 10, 2018, 4:23 PM UTC

happytoexplain wrote: post #130

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

I agree with you, but I have to point out, because it's so common: The "this will be unpopular, but I don't care" preface is, I feel, about as damaging to the perception of whatever you're about to say as "I'm not racist, but". To make an effective point, I think you should avoid, as the first thing you say, painting yourself as an underdog brave enough to speak out by preemptively criticising your audience's reactio…

Also, the "this will be unpopular" seems like it inevitably turns out to be virtue signaling for the "un-PC" people who will then aggressively make the thing popular anyway.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#149

Post by dudul » Wed, Oct 10, 2018, 4:24 PM UTC

guitarbill wrote: post #143

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ". And th…

I don't understand how your example is an evidence that data came from mean who didn't want to hire women.

These interview processes exist sure, and I personally find them idiotic, but is there evidence that they disqualify women more than men?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#150

Post by bicubic » Wed, Oct 10, 2018, 4:24 PM UTC

guitarbill wrote: post #143

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ". And th…

> One issue that keeps happening is an over-emphasis on CS-related questions

No. If you go down that path, then you are implying that women do in fact perform worse at CS-related questions. That's a much bigger can of worms than the bias being implicated here.