11–20 of 433 posts
I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
If we're being honest, a system only needs to be in a decision-making capacity for discriminatory behavior to be scrutinized, since in many cases human operators will not be able to identify the specific features being used to make decisions about people -- the features could be highly correlated with some subpopulation of protected class. If you take that to be true, the question reduces onto what decision-making roles ML algorithms have that could be discriminatory, and it's hard to argue this is not a massive part of their current and expected roles.
I think this is going to be a long, winding ethical nightmare that is probably just getting started by human-digestible examples such as these. One can imagine things like this one being looked back on as quaint in the naivety to which we assume we can understand these systems. Where do we draw the line, and how much control do we give up to an optimization function? Surely there is a balance -- how do we categorize and made good decisions around this?
As far as I know, a cohesive ethical framework around this is pretty much non-existent -- the current regime is simply "someone speaks up when something absurdly and overtly bad happens."
I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
Maybe, but it seems more likely that the model just didn't work. From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"
I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
I'm hoping industries that hire young are seeing different numbers than I did, because that should signal a shift in older ones that hire senior discipline engineers after a decade or so.
Edit: that said, companies should continue to do what they can to remediate this, but I am furious that the government has done almost nothing about the issue. The underrepresented remain exactly that.
I'm reminded of why Watson failed, and the problem with ml and ai in general- you can't peek under the hood to see why something happened, or how to keep it from happening without a lot of time, a lot of hard work, and a whole lot of carefully groomed data.
That's myth. There are approaches to analyze and debug NNs, deep dream basically fell out from one of those.
I hate this industry. Shooting themselves in the foot over and over again because no one can get passed the idea that possibly, women can be just as good at math, logic and computer science - if people would just let them. This never ends. It's just one place after another, when it gets discovered. It never changes.
Your comment will probably end up buried, but it does raise the question - if they want more female employees, was the issue in the training data, or their recruitment process?
This should be obvious when testing. Whether the algorithm discriminates should be a top priority for designing these algorithms. That's half the damn math of machine learning. If you can construct an AI, you should know how to test it for flaw in reasoning. It's just another layer of ML to do that. Outliers. It's short sighted to push these things out assuming their output is correct just because it looks 'normal'.
I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
Maybe, but it seems more likely that the model just didn't work. From the article: "Gender bias was not the only issue. Problems with the data that underpinned the models’ judgments meant that unqualified candidates were often recommended for all manner of jobs... With the technology returning results almost at random"
If there are strictly fewer women in the underlying training set, the model can still return something resembling a uniform distribution of candidates while exacerbating the diminished representation of women.
To give a concrete example: you have a bag of blue dice and red dice. There is a supermajority of blue dice in the bag. Your algorithm selects a single die out of the bag on every iteration. The output sequence of dice numbers appears uniform, but there are more blue dice than red dice in the output sequence.
I would guess that the training data for the ML set was the set of all resumes and an indicator of whether the candidate was eventually hired (maybe with supplemental data about how far in the process the candidate got). Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process?
> Could this be a direct indicator of a powerful subconscious bias in Amazon's existing hiring process? Yes, but only in the obvious sense that we all already knew: tech companies hire more men than women for technology-focused roles. That's not to say it isn't an issue; it is, but it's nothing new, and almost certainly not unique to Amazon. Without significant oversight and manual tuning, any training dataset based…