The article never mentions what form of supervision Amazon used for building this model.
I can easily see a bias towards a particular gender arising, even when a team had intentions to do, simply because of how the data is selected.
________
Case 1 : They used similarity statistics between candidates that were hired and applicants.
This is the easy one. No need to label datasets. The approach is semi-supervised.
It will also 100% cause a bias towards employees with similar profiles as those already working at Amazon. (ie. men)
________
Case 2 : They manually labelled/ranked a dataset of resumes and assigned them scores. (more likely)
Here the implicit bias of the mechanical turks / rubric would be visible. If higher scores were assigned to traditionally masculine activities, then male resumes would stand out.
I doubt this was the case though, as people at Amazon are generally competent enough to not make such a trivial mistake. Also, gender only gets mentioned in non-technical skills (mean's team, women's club, etc), which in general are not the most relevant part of the profile anyways.
________
Speculation:
1.
> Instead, the technology favored candidates who described themselves using verbs more commonly found on male engineers’ resumes, such as “executed” and “captured,” one person said.
I wonder if this has anything to do with gender at all. All good resumes that I've read use action words irrespective of gender. Maybe type A personalities use words like "captured" more often than type B ones, and the % of men in type A categories are greater. Would discrimination against women in such a case be unfair ?....Maybe.....Maybe not.
2.
Extracurricular activities are more prominent on weaker resumes than stronger ones. So, the model may be weighing down resumes with too much extracurricular fluff vs technical skills.
Men's activities are rarely prefaced with the word "men" in it. (they would just say Football team, Chess team). Women's activities on the other hand, always have the word "women" attached to it. If the extracurricular activities were penalized, then the words inside of them, including "women" would also be penalized. Thus, the model learns a latent gender bias without any bad intentions.
_________
In ML, one of my favorite statements is : "The model is only as good as the data it is trained on."
If the data is not sufficient, rich enough or prepared in the correct manner, then unintended consequences are nearly guaranteed.