Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Amazon scraps secret AI recruiting tool that showed bias against women

reuters.com

Post reply on HN
Print view View original

391–400 of 433 posts

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#391

Post by hamandcheese »

petsormeat wrote: post #156
macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Oh, there are a bunch of us, even here in the SF Bay Area. Trouble is, we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals. I could staff an entire dev team with non-male people within a week.

> I could staff an entire dev team with non-male people within a week.

I think the recruiting team at my company would very much be interested in speaking with you.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#392

Post by hamandcheese »

kat wrote: post #269
macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

"Untapped market of engineers" yes it exists. The majority of my female friends with STEM degrees ended up as high school teachers. I had several older people suggest to me that teaching should be my preferred career choice because it was more flexible than a programming job (wtf...) "every tech company would be taking advantage of it" - nope, no one is. I don't know why but my guess is its hard to admit you're doing…

Why did they choose being a high school teacher?

Are there specifics about hiring processes in tech that bias against or scare away female candidates?

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#393

Post by xedrac »

N0RMAN wrote: post #199
bicubic wrote: post #124

Earlier quoted context omitted.

Last place I worked at had to invest in additional resources to hire several females because the office was mostly male and applicants were overwhelmingly male. We did eventually find a few great female applicants, but it took a lot of work and a lot of time dedicated specifically to that goal. There does not exist some magical undiscovered pool of talented female engineers that are being turned away by biased recrui…

Just being female is a qualification at your workplace?

I worked at a company that prided itself as a meritocracy and we produced phenomenal value for our customers. We eventually got purchased by a huge corporation, which took over our hiring processes, and told us we could no longer assess a candidate's technical skill level. Soon after that, our best employees started leaving and they were being replaced with people that improved our diversity numbers, but had very little technical capability. If you were female, black, or LGBT, you were hired on the spot. Some of the female candidates were good, but the majority of the new hires were a dead weight and the productivity of the office tanked. I am all for equality, but it's sheer nonsense to hire based on gender, race, or sexual orientation.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#394

Post by Reelin »

8note wrote: post #388
ewjordan wrote: post #361

Earlier quoted context omitted.

I don't think I'm with you. A neural net cannot do this - picking apart male and female tokens requires a signal in the gradients that force the two classes apart. If there's no gradient, then something like weight decay will just zero out the weights for the "gender" feature, even if it's there to begin with. Confidence wouldn't enter in, because the feature is irrelevant to the loss function. A class imbalance does…

supposing waiter and waitress are both equally qualifying for a job, and most applicants are men, won't the ai score waiter as being more valuable than waitress?

Not generally. The entire point being made is that whether one feature is deemed to be more valuable than another feature depends not just on the data fed into the system but also on the training method used.

Specifically, the gp is pointing out that typical approaches will not pay attention to a feature that doesn't have many data points associated with it. In other words, if it hasn't seen very much of something then it won't "form an opinion" about it and thus the other features will be the ones determining the output value.

Additionally, the gp also points out that if you were to accidentally do something (say, feed in non-tech resumes) that exposed your model to an otherwise missing feature (say, predominantly female hobbies or women's colleges or whatever) in a negative light, then you will have (inadvertently) directly trained your model to treat those features as negatives.

Of course, another (hacky) hypothetical (noted elsewhere in this thread) would be to use "resume + hire/pass" as your data set. In that case, your model would simply try to emulate your current hiring practices. If your current practices exhibit a notable bias towards a given feature, then your model presumably will too.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#395

Post by blub »

petsormeat wrote: post #156
macinjosh wrote: post #93

This will be unpopular but I don't care. What is the evidence that the source data for this 'AI' is biased because the men it came from did not want to hire women? Is there a reserve of unemployed non-male engineers out there? If so what evidence is there of that? Technical talent is both expensive and a rare commodity for tech companies. The non-male engineers I've worked with have always been exceedingly competent,…

Oh, there are a bunch of us, even here in the SF Bay Area. Trouble is, we're older than 35, or don't have degrees from "top" schools, and/or don't have the "passion" for bizarre extended hiring rituals. I could staff an entire dev team with non-male people within a week.

SW engineering can be a pretty brutal job psychologically - there's a reason people are burning out. Interviewing is particularly bad.

I guess most men have thick skin or got lucky, so they don't see that, instead they think it's this dream job and everyone should partake in its wonderfulness.

I believe that the lack of women in tech is explained by societal bias against women and the nature of the job.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#396

Post by zby »

gspetr wrote: post #364
zby wrote: post #159

Earlier quoted context omitted.

"The non-male engineers I've worked with have always been exceedingly competent, smart, and their differing perspectives invaluable" - that is really an (anecdotal) evidence that there is a bias indeed. If the recruiting was all unbiased than the quality of existing male and female force would be the same - if female workers are of higher quality than it means that they need to pass higher requirements.

Another corollary should immediately follow: If an average woman in the industry is that much better than an average man, where are all the female-only companies? Does nobody want money in a capitalist society?

How much is 'that much'? The effect is probably small but what is more important it is not about a moved centre of (a Gaussian) distribution - but rather about a higher threshold usually used for women to pass.

This is of course very much dependant on the distribution shapes and I am too lazy to make a thorough analysis - but:

Let's assume that on average females were 10% more efficient programmers - but with the effort to find one female programmer you can find 10 male programmers. How much more effort do you need to find a 10% better programmer - twice as much as for the average one? Even if it was 8 times harder - then still it would make more sense to look for only men than for only women. Of course the optimal way would be to be unbiased and look for any gender.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#397

Post by zaarn »

gsich wrote: post #329
kareemsabri wrote: post #327

Earlier quoted context omitted.

They didn’t. It was discovered through other signals (mention of membership in “women’s” clubs etc.

So they did. It should be obvious that if you don't want to include gender, then you have to sanitize gender-related data.

That's not as easy as one might think.

Machine learning generally doesn't have any prior opinions about things and will learn any possible correlation in the data.

It could for example discover that certain words or sentence structures used in the resume are more likely associated with bad candidates. Later you find out that has a huge amount of people that use these certain words/structures while most other people don't.

And now the AI discriminates against them.

ML will pick up on any possible signal including noise.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#398

Post by zaarn »

chiefalchemist wrote: post #368
theptip wrote: post #125

Earlier quoted context omitted.

> "No bias" means that gender is irrelevant False. If we're talking about the technical statistical definition, bias means systematic deviation from the underlying truth in the data -- see this article by Chris Stucchio with some images for clarification: https://jacobitemag.com/2017/08/29/a-i-bias-doesnt-mean-what... "In statistics, a “bias” is defined as a statistical predictor which makes errors that all have the…

Question: So technically, the AI is not bias against women per se, but a set of characteristics / properties, that are more common among women. I'm not trying to split hairs (or argue), as much as further clarify the difference between (the common definition of) human bias and that of statistical bias.

Correct.

Computers are very bad at actually discriminating against people, they will pick up a possible bias in a statistical dataset (ie, uses certain sentence structure and is statistically less likely to get or keep the job).

Sometimes computers also pick up on statistical truths that we don't like, ie, you assign a ML to classify how likely someone is to pay back their loan and it picks up on poor people and bad neighborhoods, disproportionately affecting people of color or low income households. In theory there is nothing wrong with the data, after all, these are the people who are least likely to pay back a loan, but our moral framework usually classifies this as bad and discriminatory.

Machine Learning (AI) doesn't have moral frameworks and doesn't know what the truth is. The answers it can give us may not be answers we like or want or should have.

on a side note; human bias is usually not that different since the brain can be simplified as a bayesian filter; there are predictions on the present based on past experience, reevaluation of past experience based on current experience and prediction of future experience based on past and current experience. It's a simplification but usually most human bias is based on one of these, either explicitly social (bad experience with certain classes of people) or implicitly (tribalism).

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#399

Post by zaarn »

deegles wrote: post #236
jonny_eh wrote: post #232

Earlier quoted context omitted.

Men are promoted quicker, and more often, than women.

There was a company meeting one year at Amazon when they proudly announced that men and women were paid within 1-2% of each other for the same roles. It completely missed the point which you raise. I want to see reports of average tenure and time between promotions by gender. I suspect that the reason we don't see those published is that the numbers are damning.

Or possibly noone did a study of sufficient size that passed peer review.

It's also not hard to make the pay gap 1-2% just like it's not hard to make it 25% (both values are valid). Statistics is a fun field. Don't trust statistics you didn't fake yourself.

Amazon could easily cook the numbers to get to 1-2%, I doubt anyone checked the process of determining that number if it's unbiased and fair and accounts for other factors or not.

Re: Amazon scraps secret AI recruiting tool that showed bias against women

#400

Post by zimablue »

gspetr wrote: post #362
zimablue wrote: post #328

Earlier quoted context omitted.

I wasn't around two generations ago to make the comparison but I imagine that with the higher income has come much higher expectations that you'll be in the office for 12 hours a day and weekends. You also have much higher wealth in western nations which correlates with higher ability to seek jobs that fit your preferences. Back in the day most people didn't go to uni and had a much smaller choice of positions. There…

> Back in the day most people didn't go to uni and had a much smaller choice of positions. Not the parent and I wasn't around either, but I think accessibility of education counters your point, not supports it. More egalitarianism should in theory be more favorable to women. Back then I imagine it was much harder to program without access to university computers and education materials. More women get higher educatio…

More egalitarianism should be favorable to women /if/ you assume a priori that they are mostly disadvantaged through lack of access to education/resources, and the real expected outcome distribution is 50-50.

If you assume that there are underlying differences in interests and aptitude, more egalitarianism allows these differences to be expressed more since women are more free to eg. choose a career working with people, like medicine or law. http://www.thejournal.ie/gender-equality-countries-stem-girl... It also raises the bar for inherent aptitude to get into/(the top of) a career, since you're competing against a much wider pool of talent.

The point I was making to the parent was that his point "cross-generation drop in ratio proves it's cultural" doesn't hold up, because there have been many changes across those generations, you're comparing apples to oranges.

Post reply on HN