Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Algorithmic Monocultures in Hiring

hai.stanford.edu

Post reply on HN
Print view View original

81–90 of 177 posts

Re: Algorithmic Monocultures in Hiring

#81

Post by AStrangeMorrow »

gacgacgac wrote: post #41
wand3r wrote: post #24

Did I miss the part of the article where they break down how they determined race? Is the algorithm blind to race? It looks like they specifically looked at 83k people applying to ~100 companies which notably were Fortune 500 companies. Could there simply be candidate discrepancies here? Hard for me to follow the full methodology but it doesn't necessarily seem either malicious or that well structured. Don't you need…

Yes. You missed it. They are using a test dataset of 83k resumes generated in 2022 for this paper and comparing it as a baseline against their observational data: https://www.nber.org/papers/w29053 The dataset is constructed, deliberately, to hold candidate performance constant and vary the names of candidates to appear to be associated with a specific race.

From looking at how that was done, it seems they (the paper you linked) used an older paper which looked at which names are frequent enough and more biased toward a certain demographic (90% of that name occurrence falls within that demographic).

But they picked 9 family names per group. Which sounds quite low. And combined that with first names to reach 500 first+last names per group.

I wonder how much of the bias we see has to do with the names actually picked versus it being racially motivated (absolutely not denying that this probably is a factor, but might not be the only one).

For example, in France there is the national BAC end of high school exam. If you you at the names X grade distribution, and look at the higher “very good” bracket: some names are heavily under-represented (less than 5% of say “Jordan” get that grade) while some are over-represented (35% of “Josephine” get such a grade). The exam is for the most part anonymous, but some names are definitely heavily correlated with lower/higher income groups. So nothing surprising: Josephines tend to come from richer families, thus in average get better education/support, thus better grades. Same thing is true with family names to a smaller extent.

So I wonder how much of the bias we see, be it from real persons or the AI has more to do with a class thing than a racial thing. Again those are not neatly separate things, but still

Re: Algorithmic Monocultures in Hiring

#82

Post by 59percentmore »

paisawalla wrote: post #30
dash2 wrote: post #17

> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group That seems like a nonsensical way to measure racial discrimination. What could justify it?

This is an application of the disparate impact doctrine. Even facially neutral policies are considered suspect if they produce results that correlate against protected groups, irrespective of intent. This doctrine is the basis for much of employment law. It is a significant reason why employers don't administer IQ tests (or equivalents) to screen candidates since ~the 90s. A common objection to the doctrine is that i…

And a common rebuttal to the objection is that systemic racism is often difficult to untangle in a way that produces a neat chain of cause and effect (not least of which because discrimination can happen unconsciously or secretly); because the impact exists whether intent can be shown or not, the desire remains to ameliorate that impact.

If the issue happens upstream of the defendant to a claim - generally an organization being sued by an individual with fewer resources - it incentivizes such entities to push for changes upstream, so that they don't get stuck with the bill.

Re: Algorithmic Monocultures in Hiring

#83

Post by bakugo »

gacgacgac wrote: post #55
bakugo wrote: post #9

> To put this in perspective: If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants) Some people just can't help but put their biases on display at every opportunity, even when it comes to the most minute details.

Nothing in this has any bias in it? Which words are you suggesting are biased? This study measured constructed resumes where only names were changed, and observed the rate each group was favored (the percentage of resumes that passed). One group must be "most favored" because thats how math works. It's the group whose percentage was the highest. The resumes were fictional and equivalent across race, only the names we…

Look closer at the capitalization of the words in the quoted sentence.

Re: Algorithmic Monocultures in Hiring

#84

Post by _0ffh »

gacgacgac wrote: post #47
ApolloFortyNine wrote: post #38

I truly don't doubt it's possible for the AI to be 'racist'. >If the AI had recommended Black and Asian candidates at the same rate as it recommended the most-favored group (typically white applicants), 40,000 more of their applications would have advanced to the next stage of hiring. I don't think this is the right benchmark here, or at least, it would be very interesting if the actual outcome, offer or rejected, wa…

You are misreading this sentence. This sentence is saying: "Using a constructed dataset of resumes, whose only difference was a name change, we would anticipate a system evaluating on qualifications to produce an equal distribution of candidates across names. Our observed result was highly unequal, and that warrants further investigation."

To me it appears as if the study using the constructed dataset was a completely different one than the one that was concerned with AI.

For the AI study real data from "3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors" was used.

Re: Algorithmic Monocultures in Hiring

#85

Post by 59percentmore »

logicchains wrote: post #28
dash2 wrote: post #17

> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group That seems like a nonsensical way to measure racial discrimination. What could justify it?

>What could justify it? The assumption that applicants from all races are on average equally qualified for every position. Whole subfields of modern academia are based on that assumption.

The assumption is that no one has the authority to decide that all races aren't equally qualified for every position.

Re: Algorithmic Monocultures in Hiring

#86

Post by zerocrates »

OrvalWintermute wrote: post #72

The Pymetrics game is rigged by design: Only 40% self report gender/race no resume data, no education information, degrees, schools, GPA, major, work experience, skills/certifications Zero job qualifications

Well, they're only looking at whether the pymetrics gameplay algorithm ML thing recommends the candidate, not any of that other stuff. The outcome they're looking at here isn't whether the people actually got hired, or got passed by other screening layers or anything.

Re: Algorithmic Monocultures in Hiring

#88

Post by efavdb »

verteu wrote: post #22

The paper is here: https://arxiv.org/pdf/2605.27371 They find "disparate impact" of pymetrics across racial groups, but it doesn't seem like they controlled for anything.

They also say that if they do the analysis globally the effect goes away. Curious, does that not imply that if one domain is biased against some group there would be another where the bias was in its favor?

Re: Algorithmic Monocultures in Hiring

#89

Post by rayiner »

SiempreViernes wrote: post #75
paisawalla wrote: post #48

Earlier quoted context omitted.

You are selectively adhering to the letter of the law, when the practical effects are already well known and studied. One is not obligated to ignore literature, nor abstain from doing a simple extrapolation from the incentives placed on the table. There is a large body of literature concerning the question "does disparate-impact enforcement cause employers to alter hiring behavior in ways unrelated to actual producti…

That's not particularly surprising nor objectionable, of course legislation that reminds employers they shouldn't discriminate based on race changes practice even for companies that aren't actually caught doing it. To act like it's bad that people of colour have a more fair chance of getting employed because of some piece of legislation is simply insidious. It's just been over a month since black people lost the righ…

> It's just been over a month since black people lost the right to a fair vote.

Literally the opposite happened. The Supreme Court ruled that there was VRA §2 liability when there was evidence of racially-motivated gerrymandering: "In short, §2 imposes liability only when the evidence supports a strong inference that the State intentionally drew its districts to afford minority voters less opportunity because of their race." (Louisiana v. Callais, p. 26)

Post reply on HN