Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#151

Post by zby » Wed, Oct 10, 2018, 4:25 PM UTC

kareemsabri wrote: post #45

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

"they could be unbiased while still amassing a data set that skewed heavily male" - this sounds like a self contradiction

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#152

Post by weliketocode » Wed, Oct 10, 2018, 4:26 PM UTC

darawk wrote: post #131

jkingsbery wrote: post #52

(Disclaimer: I am an Amazon employee sharing his own experience, but do not speak in any official capacity for Amazon. I don't know anything about the system mentioned in this article.) I am a frequent interviewer for engineering roles at Amazon. As part of the interview training and other forums, we often discuss the importance of removing bias, looking out for unconscious bias, and so on. The recruiters I know at A…

It seems like they did think it through, though? And that's why it's being shelved. I don't really see what the story is here. It seems like the whole process worked exactly as it should - Amazon tried something, it had some unintended consequences, they caught it, and shelved it.

Agreed. There is no story here.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#153

Post by arandr0x » Wed, Oct 10, 2018, 4:27 PM UTC

commandlinefan wrote: post #110

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Yeah - I'm not even expressing an unpopular opinion, just asking a (leading) question: where are all these women who are chomping at the bit to get into _technical_ positions like programming but find themselves being turned away by biased recruiters? I've never even seen somebody _claim_ that they were a woman who couldn't find a tech job, just people wondering where all the women were.

They don't need to be currently unemployed. There are certainly enough women currently employed as programmers in the US to fill all the currently open Amazon positions, it's just it would be positively insane for Amazon to offer twice the salary and work-from-home to female candidates just to have prettier numbers.

Amazon (and a lot of other big non-diverse companies) are therefore hoping what is actually happening is that they are the women's first choice already, but turn them down for some reason, and that they would not have to change a thing about their work process to attract more women, except to start seeing them.

It's obvious why companies like thinking that way, and it's possible to some extent it's true. However, the fact is, if they're playing like this it's a zero-sum game and it is not actually going to improve the diversity numbers.

On my part, I've been wondering. If all these companies want to know where the women who can code are... why not just ask them? Why do you never see a "Female Programmer's Career Survey" with questions such as "On average, over the last ten years, how long have you been looking for a job?" "Would you accept a new job for a $5k raise?" "Have you ever dropped out of a hiring process because of sexism?" Take it out there in the open. Ask the real questions.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#154

Post by crooked-v » Wed, Oct 10, 2018, 4:27 PM UTC

21 wrote: post #66

komali2 wrote: post #27

Earlier quoted context omitted.

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

> There's also the idea that lack of women scientist "heroes" can be limiting (lack of role models) This one is a bit weird, computer guys were always "nerds" and "geeks" to stay away from.

...starting in the 80s, which is also when the percentage of women going into computer fields started dropping like a rock.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#155

Post by guitarbill » Wed, Oct 10, 2018, 4:29 PM UTC

bicubic wrote: post #150

guitarbill wrote: post #143

Earlier quoted context omitted.

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ". And th…

> One issue that keeps happening is an over-emphasis on CS-related questions No. If you go down that path, then you are implying that women do in fact perform worse at CS-related questions. That's a much bigger can of worms than the bias being implicated here.

Hopefully we can at least agree that those questions are limited in effectiveness, and often have no actual relation to how good an engineer is. It varies of course.

Sometimes, they seem more like a secret handshake you need to memorize to get into the boys club than actually useful engineering. Who hasn't had to revise some of these before applying for a job 3 years out of college?

What it does do is effectively exclude applicants who didn't study CS, or who haven't heard of and memorized "cracking the coding interview".

Assuming `fake CS questions == good engineer` is a huge mistake, but one i keep getting downvoted for everytime i mention it. Most rebuttals are usually something like "it's the best system we have", something i find unsatisfying.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#156

Post by petsormeat » Wed, Oct 10, 2018, 4:31 PM UTC

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Oh, there are a bunch of us, even here in the SF Bay Area. Trouble is, we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals. I could staff an entire dev team with non-male people within a week.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#157

Post by Greed » Wed, Oct 10, 2018, 4:32 PM UTC

bicubic wrote: post #150

guitarbill wrote: post #143

Earlier quoted context omitted.

> What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? One issue that keeps happening is an over-emphasis on CS-related questions. There are many great engineers I've worked with who didn't do a CS degree, and even though they are brilliant thinkers and talented engineers, too many times the interview question is "solve this problem using ". And th…

> One issue that keeps happening is an over-emphasis on CS-related questions No. If you go down that path, then you are implying that women do in fact perform worse at CS-related questions. That's a much bigger can of worms than the bias being implicated here.

It may imply this on a long enough timescale, but as it stands it only implies more that there is a historically larger pool of capable female employees sans CS degrees than there are candidates with. Which is demonstrably true.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#158

Post by astrodust » Wed, Oct 10, 2018, 4:32 PM UTC

BurningFrog wrote: post #123

kareemsabri wrote: post #45

Earlier quoted context omitted.

This doesn’t seem to be a reasonable conclusion. There is no reason to assume the AI’s assessment methods will mirror those of the recruiters. If Amazon did most of it’s hiring when programming was a task primarily performed by men, and so Amazon didn’t receive many female applicants, they could be unbiased while still amassing a data set that skewed heavily male. The machine would then just correctly assess that fem…

Control question for if you're making a certain intellectual mistake. The data set will also have skewed heavily against people named "David". Probably only ~1% of the successful applicants. Would you also expect the machine to be biased against candidates named David?

What if people named David got hired 10/100 times in the past but people named Denise only got hired 6/100 times?

Hiring practices as expressed in the data get picked up by the machine and applied accordingly. As such, David is predicted to be a better hire than Denise.

This is not about "David" vs. "Denise", but how the machine learning process will aggregate and classify names. David and David-like names will come out on top while obscure names it has no idea how to deal with (0/0 historically) will probably be given no weighting at all.

Sorry "Daud!" Our algorithm says David is better.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#159

Post by zby » Wed, Oct 10, 2018, 4:32 PM UTC

macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

"The non-male engineers I've worked with have always been exceedingly competent, smart, and their differing perspectives invaluable" - that is really an (anecdotal) evidence that there is a bias indeed. If the recruiting was all unbiased than the quality of existing male and female force would be the same - if female workers are of higher quality than it means that they need to pass higher requirements.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#160

Post by moate » Wed, Oct 10, 2018, 4:33 PM UTC

stef25 wrote: post #103

electrograv wrote: post #44

I wish we could move away from resumes for tech role screening anyway, since they convey very little real reliable information. I’ve seen too many great hires from candidates with relatively weak resumes, and failed interviews from candidates with great resumes (and obviously vice versa). I’m not sure what the best alternative should be, though. I am a fan of open source work as a sort of code portfolio, but it doesn…

> author-blind reviews Should I ever be in a position to hire a colleague, I wouldn't ever do so without having a chat with them. I spend 8hrs a day in an office with my colleagues (sometimes more than with my wife & kid) and the ones I can't stand is about the only thing wrong with my job. If we can't even see the person's face over some gender bias hysteria then I wonder how the hell we got here. People should just…

You'll also get people who (rightfully) point out that "plays nice with others" is a nice to have and not a must have. Certain people have skills so valuable they don't need to be liked (the productivity lost due to internal politics/people complaining is It's great when everyone can coexist perfectly. However, that might not be the best business decision. There's no such thing as "objectively best" just a list of pros and cons to any candidate, and a company's internal preferences.