Amazon scraps secret AI recruiting tool that showed bias against women

Unless they presented lots of unqualified resumes of people not in tech as part of the training, which seems like something someone might think reasonable. Then, the model would (correctly) determine that very few people coming from women's colleges are CS majors, and penalize them. However, I'd still expect a well built model to adjust so that if someone was a CS major, it would adjust accordingly and get rid of any default penalty for being at a particular college.

If the whole thing was hand-engineered, then of course all bets are off. It's hard to deal well with unbalanced classes, and as you mentioned, without knowing what their data looks like we can only speculate on what really happened.

But I will say this: this is not a general failure of ML, these sorts of problems can be avoided if you know what you're doing, unless your data is garbage.

The actual newsworthy part, which is getting slightly stale, is that it was influenced by the data bias.

I am not even a fan of Amazon but I think this is unfair to them. They did the right thing here.

Time and time again people (mostly men of course) keep asking "but why? why aren't there more women in the field?" Time and time again they keep saying "but I don't see any sexism in the workplace, it's nothing like it used to be, it's practically a meritocracy these days!" Yes, indeed, it truly is a giant mystery.

And yet, at the same time there is a constant deluge of stories about rampant sexism in the industry. Of all sorts, at all levels, at almost every company, and often of shockingly regressive character even up through the present time. There are countless stories in the industry of how women in tech are persistently denigrated, how men talk over them in meetings, how their ideas are ignored until they come out of the mouth of a man, how sexual harassment is ubiquitous, how they are routinely excluded from workplace culture through extremely male-centric activities that include things as ridiculous as morale events or even meetings held at strip clubs.

All of this takes a toll, and that toll is ultimately to stunt the careers of women in tech and to push women out of the industry entirely. Working in tech as a woman is climbing a hill with a much steeper slope than it is for guys. Women routinely get passed over for promotions, are routinely underpaid, routinely do not receive credit for their ideas, and routinely experience more hostile working conditions (through bias as well as sexual harassment). So they leave. They find something better to do with their time because they just can't take the stress and harassment anymore or because it just does not provide the same return on investment as it does for guys.

And we know this. We know this from studies and exposes and a torrent of anecdotes from individual women who have been in the field for years or decades. Some people (guys) have a tendency to write off each and every one of these stories and studies as somehow individual aberrations or outliers which don't have any bearing on the fundamental overall character of the industry, but this is a mistake, they are absolutely representative. The problem of over-representation of white men in tech cannot be solved by "fixing the pipeline" in the educational system nor can it be solved by making hiring processes perfectly unbiased (or even biased towards women) because the real problem is much bigger, it's systemic, widespread misogyny throughout the entire industry. That will take a tremendous amount of work to fix, but once the industry stops treating women as second class citizens (or exotic outsiders) and stops pushing them out of the industry through its toxicity then the problem will mostly fix itself.