Amazon scraps secret AI recruiting tool that showed bias against women • Hacker News Board

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#321

Post by beat » Wed, Oct 10, 2018, 9:23 PM UTC

axlprose wrote: post #316

beat wrote: post #264

Earlier quoted context omitted.

There's a presupposition that the natural distribution of intelligence is gender-neutral. Which suggests that the unequal distribution of software engineers by gender has a cause other than intelligence. So what is the cause, then? Is it biological, or social, or random chance? "Random" doesn't seem likely, especially given how many other professions are male-dominated, and the relative economic and social power of t…

is there a reason why you're so intently focused on the metric of intelligence here, as if it's the end-all-be-all of psychological factors? I work in personality psychology research, so this whole IQ-centric line of reasoning is very dubious to me. There are many other influential phycological factors involved in people's lives that aren't (as far as we know) a direct result of nurture, and when taken together often…

It doesn't need to be IQ-based. I'm dubious about any sort of "genetic" argument for why some fields are dominated by men, and others by women. The shift in programming from primarily women to primarily men is evidence for that, imho - if the leanings are genetic, why a change over the course of one or two generations?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#322

Post by fizwhiz » Wed, Oct 10, 2018, 9:27 PM UTC

jonny_eh wrote: post #232

gambler wrote: post #224

Earlier quoted context omitted.

The article didn't specify how they labeled resumes for training. You're assuming that it was based on whether or not the candidate was hire. Nobody with an iota of experience in machine learning would do something like that. (For obvious reasons: you can't tell from your data whether people you did not hire were truly bad.) A far more reasonable way would be to take resumes of people who were hired and train the mod…

Men are promoted quicker, and more often, than women.

citation needed

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#323

Post by petsormeat » Wed, Oct 10, 2018, 9:44 PM UTC

jhfhhhf wrote: post #287

petsormeat wrote: post #156

Earlier quoted context omitted.

Oh, there are a bunch of us, even here in the SF Bay Area. Trouble is, we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals. I could staff an entire dev team with non-male people within a week.

So what do you do now? Btw some men in tech are also over 35 and tired of hiring rituals.

After 15 years of front-end dev, I now work in retail. Some of my other peers are scraping by with Uber/Lyft. Some are muddling through as housewives or substitute teaching.

And, yes, Bay Area tech hiring is needlessly hostile for men over a certain age as well.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#324

Post by dheera » Wed, Oct 10, 2018, 9:44 PM UTC

fuscy wrote: post #25

The eye opening thing here is not that the AI failed, but why it failed. At start the AI is like a baby, it doesn't know anything or have any opinions. By teaching it using a set of data, in this case a set of resumes and the outcome then it can form an opinion. The AI becoming biased tells that the "teacher" was biased also. So actually Amazon's recruiting process seems to be a mess with the technical skills on the…

The term "AI" is over-hyped. What we have now is advanced pattern recognition, not intelligence.

Pattern recognition will learn any biases in your training data. An intelligent enough* being does much more than pattern recognition -- intelligent beings have concepts of ethics, social responsibility, value systems, dreams, ideals, and is able to know what to look for and what to ignore in the process of learning.

A dumb pattern recognition algorithm aims to maximize its correctness. Gradient descent does exactly that. It wants to be correct as much of the time as possible. An intelligent enough being, on the other hand, has at least an idea of de-prioritizing mathematical correctness and putting ethics first.

Deep learning in its current state is emphatically NOT what I would call "intelligence" in that respect.

Google had a big media blooper when their algorithm mistakenly recognized a black person as a gorilla [0]. The fundamental problem here is that state-of-the-art machine learning is not intelligent enough. It sees dark-colored pixels with a face and goes "oh, gorilla". Nothing else. The very fact that people were offended by that is a sign that people are truly intelligent. The fact that the algorithm didn't even know it was offending people is a sign that the algorithm is stupid. Emotions, the ability to be offended, and the ability to understand what offends others, are all products of true intelligence.

If you used today's state-of-the-art machine learning, fed it real data from today's world, and asked it to classify them into [good people, criminals, terrorists], you would result in an algorithm that labels all black people as criminals and all people with black hair and beards as terrorists. The algorithm might even be the most mathematically correct model. The very fact that you (I sincerely hope) cringe at the above is a sign that YOU are intelligent and this algorithm is stupid.

*People are overall intelligent, and some people behave more intelligently than others. There are members of society that do unintelligent things, like stereotyping, over-generalization, and prejudice, and others who don't.

[0] https://www.theverge.com/2018/1/12/16882408/google-racist-go...

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#325

Post by sonnyblarney » Wed, Oct 10, 2018, 9:49 PM UTC

bicubic wrote: post #74

patwolf wrote: post #42

The explanation seems overly simplistic. If the difference in volume of male candidates mattered, then I would also expect to see a bias in favor of applicants from larger universities. That seems like too obvious an issue in the way the algorithm was designed. I see four possibilities here: 1. The algorithm was designed in a completely inept fashion 2. The algorithm design was sound, but ultimately ineffective 3. Th…

We're talking about Amazon, one of the biggest powerhouse ML employers. I don't buy that the model was poorly designed or ineffective. They also didn't just scrap the model without understanding how or why it failed to meet its objectives. And whatever the cause was, it was not the poor quality of the training data. They tried to stop the model from downranking women based on obvious keywords, only to find it learnin…

"I don't buy that the model was poorly designed or ineffective."

Just because some groups have competencies in this area, doesn't mean that others do. I've worked at big tech companies that couldn't get their HR systems to work properly ... IT was abysmal even though we made 'high tech'. Also, it's an internal project, not a product, so the scope of investment etc. might have been very different than otherwise.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#326

Post by Faark » Wed, Oct 10, 2018, 9:50 PM UTC

HashHishBang wrote: post #117

Hold on here. This article seems to have buried a pretty important piece of information wayyy down in the middle of the text. > Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs, the people said. With the technology returning results almost at random, Amazon shut down the project, they said.…

The article leaves a lot open to interpretation, including what was expected of the tools... that could range from providing some beneficial hints to replacing all hiring. As you rightfully remark, having the tool rank candidates for highly specific jobs and its tech requirements would be a great achievement. But is also a big challenge, thus they probably were aiming at something more basic initially. Building models for broad categories like "manager" or "box packer" and hope they will detect soft skills or work ethics seems more achievable. Thus the additional star rating that can be used for hiring and provides some value.

Now having known limited capabilities isn't great. But those can and will be worked on. Unknown / unexpected biases wont, making finding them important.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#327

Post by kareemsabri » Wed, Oct 10, 2018, 9:51 PM UTC

gsich wrote: post #314

was_boring wrote: post #253

Earlier quoted context omitted.

There are a few ways you can tackle this issue: 1) have the same algorithm for each group, but train separately (so in the end you have two different weights); 2) over-sample the group under represented in the data; 3) make the penalty more severe for guessing wrongly on female then male applicants during training; 4) apply weights to gender encoding; 5) use more then just resumes as data. This isn't an insurmountabl…

Or: don't include gender in the training data.

They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#328

Post by zimablue » Wed, Oct 10, 2018, 9:51 PM UTC

beat wrote: post #280

zimablue wrote: post #228

Earlier quoted context omitted.

But you're not comparing the same thing generation to generation. The job of developer has changed massively, the number of developers, the expectations, the salary. Sociaety has also changed, and not just in culture but in income distribution, etc etc. If tomorrow we say that you have to do 30 chinups to be a waitress, and the job will involve regular fistfights then we count the number of waitresses by gender and s…

"The job of the developer has changed massively". What, pray tell, has changed that made the job more attractive to men, and less attractive to women? You need to be able to answer that question if you're going to make a causal assertion.

I wasn't around two generations ago to make the comparison but I imagine that with the higher income has come much higher expectations that you'll be in the office for 12 hours a day and weekends. You also have much higher wealth in western nations which correlates with higher ability to seek jobs that fit your preferences. Back in the day most people didn't go to uni and had a much smaller choice of positions. There are hundreds of ways the world and job are very different, and you're flipping the argument to say that I have to assert the one specific causal link. If you're proposing an argument "it's misogynist culture, as evidence compare two generations ago". Then it's more the case that you need to demonstrate that the conditions and job are the same, for your link to be valid. Or that all the ways they're different are irrelevant, which they're just obviously not.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#329

Post by gsich » Wed, Oct 10, 2018, 9:56 PM UTC

kareemsabri wrote: post #327

gsich wrote: post #314

Earlier quoted context omitted.

Or: don't include gender in the training data.

They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.

So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#330

Post by trustmath » Wed, Oct 10, 2018, 9:56 PM UTC

sakopov wrote: post #259

trustmath wrote: post #86

Earlier quoted context omitted.

Please read the article. This article is about automated reasoning that discards resumes that are strongly correlated to resumes of women.

I think you need to read the comment again. The author's reply was to your comment saying "if people would just let them", not what the ML algorithm does in the article.

That's somewhat appropriate, but, I think having higher standards for identifying discriminatory practices is covered under the umbrella of 'if people would just let them'.

Achieving that level of a standard is a balance.

There shouldn't be excuses being made. All that can do is contribute to the perpetuation of the conditions that presently exist, because the core issue isn't being identified.

Furthermore, if the core issue is the excuse itself, then again, this is covered under the umbrella of 'if people would just let them'. The secondary issue would then be that the core issue isn't being questioned.