Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#311

Post by monochromatic » Wed, Oct 10, 2018, 8:47 PM UTC

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

> The AI becoming biased tells that the "teacher" was biased also.

That doesn’t follow.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#312

Post by keithnoizu » Wed, Oct 10, 2018, 8:53 PM UTC

dx87 wrote: post #277

MetalGuru wrote: post #234

Earlier quoted context omitted.

What came first? Was the cast of Revenge of the Nerds all males because nerds were males and art was depicting reality, or did only men start becoming nerds become this movie had an all-male cast? I get being envious of people who had a head start with computers and feeling behind. I had zero coding experience before my first CS class in college. It sucked, but I grinded through it. It's relatively easy to catch up w…

I'll start by saying that I agree that determination can overcome a late start, but familial and society pressures can be really difficult to overcome. My wife comes from a small southern town and wanted to be a writer and college professor, but was constantly ridiculed and chastized by teachers and family because she is a woman, so they said she should just be a nurse, or get married and be a housewife. Her family r…

Supporting document: https://www.smithsonianmag.com/smart-news/computer-programmi...

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#313

Post by stevenwoo » Wed, Oct 10, 2018, 8:59 PM UTC

nv-vn wrote: post #176

komali2 wrote: post #27

Earlier quoted context omitted.

Ah, the Damore argument. Besides the fact that his psuedo science has been summarily handled[0], to consider his argument you then have to equally consider the possibility of sexism in academia pressuring women to not study these subjects and societal pressure their whole lives pressuring them to not persue these career paths. There's also the idea that lack of women scientist "heroes" can be limiting (lack of role m…

Where was it disproven? The article says that the research was controversial, not that it's false.

Notably, Damore makes pretty much the same arguments against using race in hiring as he does gender, but failed to provide any proof for his arguments, he only really gave what he interpreted as evidence for his gender beliefs. There's little to disprove except for Damore's interpretation of results as being proof for his argument.

When right wing trolls attacked a female CS lecturer, she wrote a long response here: https://www.vox.com/the-big-idea/2017/8/11/16130452/google-m...

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#314

Post by gsich » Wed, Oct 10, 2018, 9:00 PM UTC

was_boring wrote: post #253

bcheung wrote: post #239

Earlier quoted context omitted.

How did you control for these things? Wondering what patterns there are that people use to prevent social discrimination. Seems challenging since much of AI, especially classification, is essentially a discrimination algorithm.

There are a few ways you can tackle this issue: 1) have the same algorithm for each group, but train separately (so in the end you have two different weights); 2) over-sample the group under represented in the data; 3) make the penalty more severe for guessing wrongly on female then male applicants during training; 4) apply weights to gender encoding; 5) use more then just resumes as data. This isn't an insurmountabl…

Or: don't include gender in the training data.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#315

Post by noetic_techy » Wed, Oct 10, 2018, 9:02 PM UTC

gambler wrote: post #224

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

The article didn't specify how they labeled resumes for training. You're assuming that it was based on whether or not the candidate was hire. Nobody with an iota of experience in machine learning would do something like that. (For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.) A far more reasonable way would be to take resumes of people who were hired and train the mod…

I'm not a ML guy, but reading this, it almost sounds like the training data needs to be a fictional, idealized set, and not based on real world data that already has bias slants built in. Possibly composites of real world candidates with idealized characteristics and fictional career trajectories. Basically, what-my-company-looks-like vs what-I-want-it-to-look-like. I'm not sure this is even possible.

Its an interesting questions. On one hand, a practical person could argue: "Well, this is what my company looks like, and these are the types of people who fit with our culture and make it, so be it. Find me these types of candidates."

VS

"I don't like the way may company culture looks, I would rather it was more diverse. This mono-culture is potentially leaving money on the table from not being diverse enough. I'm going to take my current employees, chart their career path, composite them (maybe), tweak some of the ugly race and gender stats for those who were promoted, and feed this to my hiring algorithm."

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#316

Post by axlprose » Wed, Oct 10, 2018, 9:06 PM UTC

beat wrote: post #264

hnbroseph wrote: post #233

Earlier quoted context omitted.

> And that's where hands come up empty. there seems to be a presupposition here that the the 'natural' proportion of women software engineers is 50%.

There's a presupposition that the natural distribution of intelligence is gender-neutral. Which suggests that the unequal distribution of software engineers by gender has a cause other than intelligence. So what is the cause, then? Is it biological, or social, or random chance? "Random" doesn't seem likely, especially given how many other professions are male-dominated, and the relative economic and social power of t…

is there a reason why you're so intently focused on the metric of intelligence here, as if it's the end-all-be-all of psychological factors?

I work in personality psychology research, so this whole IQ-centric line of reasoning is very dubious to me. There are many other influential phycological factors involved in people's lives that aren't (as far as we know) a direct result of nurture, and when taken together often make a more significant contribution to people's lives than their score in the single dimension of IQ. Learning disabilities and affective/mood disorders are a big example of this, and personality traits are just as impactful in how a person's life unfolds, regardless of intelligence.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#317

Post by ergothus » Wed, Oct 10, 2018, 9:19 PM UTC

noetic_techy wrote: post #315

gambler wrote: post #224

Earlier quoted context omitted.

The article didn't specify how they labeled resumes for training. You're assuming that it was based on whether or not the candidate was hire. Nobody with an iota of experience in machine learning would do something like that. (For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.) A far more reasonable way would be to take resumes of people who were hired and train the mod…

I'm not a ML guy, but reading this, it almost sounds like the training data needs to be a fictional, idealized set, and not based on real world data that already has bias slants built in. Possibly composites of real world candidates with idealized characteristics and fictional career trajectories. Basically, what-my-company-looks-like vs what-I-want-it-to-look-like. I'm not sure this is even possible. Its an interest…

> the training data needs to be a fictional, idealized set, and not based on real world data that already has bias slants built in

Thatd be great, but in this case (as in most ML cases) the idea is not "follow this known, tedious process" but instead "we have inputs and results but dont know the rules that connect them, can you figure out the rules?"

> this is what my company looks like

In tech hiring, no one wants the team they have...they want more people but without regrets (including regretting the cost)

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#318

Post by roenxi » Wed, Oct 10, 2018, 9:19 PM UTC

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

Do you have some information not present in the article? There seem to be some assumptions on the training process in your comment that are not sourced in the article.

I'll don my flack jacket for this one, but based on population statistics I believe a statistically significant number of women have children. A plausible hypothesis is that a typical female candidate is at a 9 month disadvantage against male employees and that that is a statistically significant effect detected by this Amazon tool.

Now, the article says that the results of the tool were 'nearly random', so that probably wasn't the issue. But just because the result of a machine learning process is biased does not indicate that the teacher is biased. It indicates that the data is biased, and bias always has a chance to be linked to real-world phenomenon.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#319

Post by erik_seaberg » Wed, Oct 10, 2018, 9:21 PM UTC

beat wrote: post #267

naasking wrote: post #238

Earlier quoted context omitted.

> But there are fewer women in software now than there were 30 years ago. That's not true. Firstly, it's more like 40-50 years ago. Secondly, there are far more women doing software development, but the gender ratio is dramatically different. Thirdly, that's because male interest exploded with the advent of personal computing the 80s. Lastly, "programming" as a profession used to be regarded as an offshot of secretar…

Okay, so TWO generations. Big deal. It still dismisses the "born that way" nonsense argument. "'programming' as a profession used to be regarded as an offshoot of secretarial work, which was dominated by women". Which begs the question of why women dominated secretarial work (and still do), while as programming became a more respected and better paying profession, it became male-dominated.

"Programmer" used to be the title that goes with using a keypunch to turn a flowchart into a deck to submit to the operator. That job had low status because it sucked, for the same reason that spending all day typing someone else's words sucked. Eventually we could afford to automate that job away. "Systems analyst" and "programmer/analyst" are the titles for independent design work we should be comparing to today's developers.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#320

Post by noetic_techy » Wed, Oct 10, 2018, 9:23 PM UTC

The problem is you cant feed the ML algorithm training data based on what your company currently looks like, you have to feed it an idealized set of what you want it to look like. It almost needs to be fictitious training data to hide the ugly bias that's already built in.

I don't think this will ever work. There is too much variability in resume wording that correlates to gender and even culture of origin even when you take out names and any other protected class identifying markers. The Dutch tried this and ended up with less diversity.

I'm going to go out on a limb and say you almost want to leave all that identifying data in, but put each candidate into buckets with separate rating algorithms trained against only that "type" of candidate. The top candidates from each culture, and the top candidates from each gender, etc etc, however you want to do it. Feed them into a picking algorithm that builds a composite of what you want your team to look like diversity wise based on the top candidates from each bucket, and go from there.

Don't take my opinion seriously, I'm not an ML guy.