Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

31–40 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#31

Post by hkai »

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

How did you come to the conclusion that gender was being the most important, rather than skills or aggressiveness?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#32

Post by adventured »

kgwgk wrote: post #13

I’m curious: how is it that this submission has not “reused” the submission I made eight hours before? In my experience, sending a link that is already there simply upvotes the existing submission.

After a certain amount of time, the same link can be posted by a different user. I'm not sure if there is a point or point vs time qualification on that. You can post old, popular stories again for example.

Your submission was made 11 hours ago. If they merely applied an upvote to that existing story after 10 hours, the point time value rot would be such that it would have greatly reduced impact at ranking the story to the front page (ie it would be nearly useless for discovery purposes).

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#33

Post by kaitai »

hkai wrote: post #31
fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

How did you come to the conclusion that gender was being the most important, rather than skills or aggressiveness?

I don't think that's what the parent was claiming; the parent says "gender and aggressiveness" were most important and skills listed on the resume as providing such an unclear signal for actual hires that they were not picked up by the AI.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#34

Post by sanxiyn »

WalterSear wrote: post #14
KevinEldon wrote: post #10

Earlier quoted context omitted.

Maybe, but it seems more likely that the model just didn't work. From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"

Sounds like they've successfully emulated their real world interview process.

Indeed.

I am quite curious about details of the model. For example, the single largest contribution to real world interview process variability is interviewer (for resume screening, who screened that resume, etc.). Wouldn't it be possible to code interviewer as categorical variable and separate resume-intrinsic? effect and interviewer effect? They must have tried this, haven't they?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#35

Post by ummonk »

mcherm wrote: post #28
ummonk wrote: post #4

Machine learning should not be used in this way on humans, whether for resume screening or even more dystopic, for sentencing.

Why not? Seriously: let us take as given that the AI models are biased. Will you also admit that the existing processes are biased? If so, then what we need to ask is which is MORE biased. It might be complaining that we shouldn't release self-driving cars because on rare occasions they cause accidents. There is, however, another criterion besides how biased it is: how biased it will be in the future. Human-driven pr…

I think we basically have the same view. I don't support black box machine learning models. I do support using automated tests and simple well defined objective criteria though, which is basically a transparent AI model.

It's just that generally what seems to distinguish whether something is called "machine learning" rather than "data science and modelling" is that the former is black box and the latter is not.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#36

Post by Novashi »

I think there's also a subtle point here that an AI figured out that Amazon hiring was biased more quickly than Amazon itself.

I wouldn't go so far to say that running your history through an AI is necessarily proof of anything though (esp. in court). Imagine using another company's historical data to suggest that they're discriminatory.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#38

Post by sanxiyn »

michaelt wrote: post #29
ncallaway wrote: post #3

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Maybe - but maybe not. Imagine a company with 2 men in HR, 2 women in HR, 40 men in engineering, and 10 women in engineering. That's with gender-blind hiring, reflecting only the 4:1 ratio of male to female CS graduates. If you picked a random male hire, there's a 40/42=95% chance they're an engineer whereas if you p…

Why would one train on "all hires CVs"? It'd be "engineering CVs", moreover it'd be "engineering applicants CVs", not "engineering hires CVs".

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#39

Post by Spooky23 »

throwawaymath wrote: post #7
ncallaway wrote: post #3

I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?

> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based…

I spent a number of years working in tech organizations where the male:female ratio was 60:40. Now I work in a similar org with the same rules where it's more like 85:15.

The difference is that old place transitioned administrative staff to IT roles in the 90s/early 2000s when more things were computerized. Those admins, financial analysts, program analysts were more likely to be female and had degrees in liberal arts, accounting, business/finance, etc.

In the newer place, they filtered based on computer-related degree upon hiring. That automatically excludes many women. Once hired, female candidates advance as well or better.

Anecdotally, I've hired interns in recent years with no tech-specific qualifications as an experiment. If you select for "smart and gets things done" I don't see much of a disadvantage for many roles. You get some duds too, but it wasn't as dramatic a difference as I expected.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#40

Post by lalaland1125 »

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

> The number of women and men in the data set shouldn't matter (algorithms learn that even if there was 1 woman, if she was hired then it will be positive about future woman candidates).

This is incorrect. The key thing to keep in mind is that they are not just predicting who is a good candidate, they are also ranking by the certainty of their prediction.

Lower numbers of female candidates could plausibly lead to lower certainty for the prediction model as it would have less data on those people. I've never trained a model on resumes, but I definitely often see this "lower certainty on minorites" thing for models I do train.

The lower certainty would in turn lead to lower rankings for women even without any bias in the data.

Now, I'm not saying that Amazon's data isn't biased. I would not be surprised if it were. I'm just saying we should be careful in understanding what is evidence of bias and what is not.

Post reply on HN