Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Algorithmic Monocultures in Hiring

hai.stanford.edu

Post reply on HN
Print view View original

121–130 of 177 posts

Re: Algorithmic Monocultures in Hiring

#121

Post by lazide »

dash2 wrote: post #17

> To measure adverse impact, we apply the EEOC’s “four-fifths rule,” which flags a position when one group is recommended at less than 80% of the rate of the most-recommended group That seems like a nonsensical way to measure racial discrimination. What could justify it?

‘Every one is the same’, even when one group or another doesn’t like doing some kind of work for some reason.

Because surely no one would have legitimate preferences based on their gender, cultural norms, etc. or real differences in aptitude due to childhood exposure, education, or said norms and preferences.

Re: Algorithmic Monocultures in Hiring

#122

Post by pc86 »

alain94040 wrote: post #4

The European Union passed The Artificial Intelligence Act, which classifies: High-risk – AI applications that are expected to pose significant threats to health, safety, or the fundamental rights of persons. Notably, AI systems used in health, education, recruitment, critical infrastructure management, law enforcement or justice. They are subject to quality, transparency, human oversight and safety obligations That's…

This is one of those things where the first sentence sounds completely fine and reasonable, maybe even objectively good.

Of all the things listed "recruitment" doesn't belong to me. Is the argument that it is someone's fundamental human right to get someone else to pay them to do a job? Or is it strictly about human oversight?

Re: Algorithmic Monocultures in Hiring

#123

Post by tbrownaw »

> We find that people who submit multiple applications to positions screened by the same algorithmic hiring vendor are more likely to be rejected from every position to which they apply than would be true if the companies made decisions statistically independently from one another. Ten percent of applicants who submit four applications are rejected from all the places to which they apply.

> Our research also found that this pattern does not appear to be the case in other circumstances. We analyzed data from the largest prior study of hiring decisions, which sent 83,000 applications to 108 Fortune 500 firms during the same time period as our study and did not focus on whether AI was used to make decisions. We found that the rate at which applicants were rejected from every firm they applied to in this data was no higher than what you’d expect if each company decided independently of the others.

It sounds like this study was using real-world applicants, and the other study they're comparing against was using synthetic applicants.

Consider the chance of being accepted as being composed of signal+bias+noise. Noise is random. Signal is a per-applicant value, and what's meant to be measured. Bias is a per-group value, and an artifact of the measuring process.

If acceptance/rejection is independent between positions applied for (as in the synthetic applicant study), that suggests that it's random or composed entirely of noise; ie there is no signal; ie the applicants are all equally qualified.

If acceptance/rejection is correlated, that means there is some nonzero amount of (signal+bias). But real-world applicants are not all identical, so there should be some amount of signal. So you can't just assume zero signal in order to infer that there must be bias.

Re: Algorithmic Monocultures in Hiring

#124

Post by pc86 »

paisawalla wrote: post #92
59percentmore wrote: post #82

Earlier quoted context omitted.

And a common rebuttal to the objection is that systemic racism is often difficult to untangle in a way that produces a neat chain of cause and effect (not least of which because discrimination can happen unconsciously or secretly); because the impact exists whether intent can be shown or not, the desire remains to ameliorate that impact. If the issue happens upstream of the defendant to a claim - generally an organiz…

What evidence would disprove the claim that systemic racism is the cause of a persistent disparity?

Why is this the one time someone is expected to disprove a claim rather than the claimant being expected to provide evidence?

If you're making the claim you need to provide the evidence.

Most people would say that a persistent disparity means it's possible there is discrimination, but it's not definitive proof.

Re: Algorithmic Monocultures in Hiring

#125

Post by daft_pink »

pc86 wrote: post #111
daft_pink wrote: post #104

Anyone who’s done hiring wouldn’t be shocked by this: We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions. Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, m…

> a rejected resume is more likely to be rejected by every other employer This makes sense to me, albeit intuitively and in a way I can't articulate. > an accepted resume is more likely to be accepted by every other employer but this doesn't necessarily follow from the prior for me. Plenty of people get really good jobs and are really successful in them only after dozens or hundreds of rejections with a nearly-identi…

If you look at the chart, the systemic rejection rate is only like 5-10%. It’s not a huge impact and it’s just about getting an interview not getting the job, so they could still get rejected.

I just think certain resumes will get an interview almost every time in some industries and certain resumes will likely never get an interview almost every time, but the majority of resumes are like you say have different aspects that appeal to one empoyer over another.

Re: Algorithmic Monocultures in Hiring

#126

Post by tbrownaw »

kenjackson wrote: post #105

I think this partially buries the lede: "As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out." If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not b…

> We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning.

I don't think this even requires any bias.

Assume there's some loose ordering of who is or isn't a good hire, and every employer has their own fuzzy view of it. If you get slightly better or worse as a potential hire (pick up an extra degree, let your latest certification lapse, whatever), it gets somewhat easier or harder to get hired.

Now assume that same ordering, but all employers share the same view of it. I'd expect the divide between employable and not employable to be much sharper.

Re: Algorithmic Monocultures in Hiring

#127

Post by runako »

paisawalla wrote: post #112
runako wrote: post #70

Earlier quoted context omitted.

> selectively adhering to the letter of the law Are you suggesting that companies should violate the law here? What do you recommend? Edit: charitably, "adhering to the letter of the law" is sometimes shortened to "law-abiding" and is generally what we want.

You've misunderstood the point. Prior to the beginning of your excerpt is the word "You", meaning the comment's author is the subject, not "companies". I'm saying the commenter is appealing to black letter law for the answer to the question "what happens when..." but we have observational evidence to answer the question.

> we have observational evidence to answer the question.

Isn't the point that the observational evidence amounts to the companies in question steer clear of illegal behavior?

There are anti-money laundering laws, so banks institute procedures to help them comply. Yes, we expect companies to change their processes so they don't break the law. That's the point of the law.

I am confused with what you think companies should do in this situation. Expose themselves to legal and civil liability? Or change their behaviors so that close scrutiny indicates they are trying to comply with the laws and any bad actors acted against internal procedure?

Re: Algorithmic Monocultures in Hiring

#128

Post by actionfromafar »

tbrownaw wrote: post #126
kenjackson wrote: post #105

I think this partially buries the lede: "As a single hiring vendor comes to dominate screening for an industry, it may be more likely that candidates are shut out." If we move to using just a small number of AI models to help do things like hiring, we will amplify biases and possibly completely lock out portions of the population. We need to be very careful when using AI systems to evaluate people in general -- not b…

> We need to be very careful when using AI systems to evaluate people in general -- not because they might be biased (which they might be), but because even a small bias, if used by virtually everyone, can be damning. I don't think this even requires any bias. Assume there's some loose ordering of who is or isn't a good hire, and every employer has their own fuzzy view of it. If you get slightly better or worse as a…

Well, I'd say that specific ordering is the bias. But I see what you mean. The bias is arbitrary, but still very real.

Also, we will of course have all kinds of attempts to "game" the system to get ahead. Optimizing (even more) for the metric. Degree mills, for instance.

Re: Algorithmic Monocultures in Hiring

#129

Post by jongjong »

zeroonetwothree wrote: post #115
jongjong wrote: post #101

I think the discrimination aspect is downstream from this fact: > We follow 3.4 million people who submit 4 million job applications to 1,700 job postings across 150 employers and 11 industry sectors. Each job application was assessed by an AI hiring tool built by a single third-party vendor. 3.4 million people applying to just 150 employers... Who are all using just 1 platform. WTF. This is where the discrimination…

That’s the platform that gave them the data. I don’t think they claim it’s all the applications of this set of people.

I realise this but it's still incredible to think because that's about 22k applicants per company.

Even if that's just part of each company's total hiring pipeline, it's clear; something's wrong. I don't know how long this study has been running but 22k is a lot of people, even over a year. These companies are too big. That's the problem.

Re: Algorithmic Monocultures in Hiring

#130

Post by verteu »

daft_pink wrote: post #104

Anyone who’s done hiring wouldn’t be shocked by this: We find applicants are more likely to be rejected from every position they apply to than would be predicted by the baseline of each position making statistically independent decisions. Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer. Like online dating, m…

> Obviously a rejected resume is more likely to be rejected by every other employer and an accepted resume is more likely to be accepted by every other employer.

But that wasn't the case for non-algorithmic screening. From the paper:

"By contrast, we find that when first round screening is not mediated by a single screening procedure, systemic rejections are close to the baseline. To support the empirical validity of our baseline, we study homogeneous outcomes in the largest study of first-round screening at U.S. employers to date. Kline et al. [38] generated 83000 synthetic resumes and submitted these resumes to vacant positions at 108 US companies between October 2019 and April 2021, a similar time period to our data. The companies, which are a subset of the Fortune 500,15 collectively employ 15 million workers. We analyze the homogeneity observed in the resulting callback outcomes in their data. We find that the baseline is an effective estimator of the systemic rejection rate for this dataset. As shown in Figure 3, the observed systemic rejection rate is accurately predicted by the baseline and a chi-squared goodness-of-fit test cannot reject equality of the two distributions (2 = 20.05, = 0.69). In other words, while the largest previous study observes systemic rejection rates consistent with employers making statistically independent decisions, the algorithmic hiring data shows significantly correlated outcomes that lead to higher-than-baseline systemic rejection rates."

Post reply on HN