Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

arxiv.org

Post reply on HN
Print view View original

171–180 of 190 posts

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#171

Post by delusional »

lunchbucket wrote: post #163
delusional wrote: post #94

Earlier quoted context omitted.

I doubt it since they, admittedly, didn't read it. The question he posed, about the paper, is answered in that very same paper. He has structured his whole reply to have the tone of uncovering the hidden caveat in the small print that invalidates the paper, when it's actually a straightforwardly stated assumption in their methodology section.

Now that they've confirmed that was in fact what they meant, how have your views on this exchange changed?

> how have your views on this exchange changed?

Not at all, because I am critiquing the authors writings, and for those I don't need to speculate on his intentions. He wrote a comment where he misrepresents the arguments in the paper, while explicitly saying he didn't bother to read it. That's not good enough.

The author of said comment now comes in, after getting criticized, and claims that "yes, I meant that all along" and appends a note about not considering it "much" of a justification. He did not question the justification of the paper, his claim was "I can’t come up with a justification" implying the paper has NO justification for the design. His criticism of the abstract as not covering the design of the experiment rings hollow when he can't be bothered to read the paper itself.

That being said, I am happy that he went back and read the justification, and I do think it's valid to question the conclusions drawn from the design of the study. I too wonder if this result would replicate had the models been provided the entire resume. I too think presenting the model with the entire reconstructed resume would have been a stronger test.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#172

Post by qwytw »

rogermarley wrote: post #41

I think resumes will eventually (or have already) become obsolete in tech. The SNR is so low, they offer very thin filtering value. Even taking the tiny bits of the resume that are "hard signal", like GPA, certifications, prior roles, etc, it doesn't translate into their performance in the initial screening interview. This is why what I think the industry sorely needs is examination consortia. Rather than trying to g…

Maybe just a lottery instead? Would be approximately as useful just way simpler.

Also don't all of the "enterprise" certificates already provide all that, anyway?

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#173

Post by delusional »

hyperpape wrote: post #96
nearbuy wrote: post #53

Earlier quoted context omitted.

I assume they meant they can't come up with a reasonable justification.

Thank you, that's correct. To be perfectly clear, I understand their justification for only _editing_ the executive summary, it is arguably reasonable, because editing the work history would risk altering the details in ways that compromise the measurement. This is a hard problem to solve (you might try reviewing the resumes for hallucinations, but I can't think of a precise study design that doesn't risk problems).…

> Worse, this isn't just "decent paper, bad summary", their abstract misreports their findings.

What findings are being misrepresented? Their claims seem supported by their conclusions to me. You can question the generality of their claims based on the limitation of their methods, but that does not amount to "misreporting" the conclusion.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#174

Post by 3eb7988a1663 »

Melatonic wrote: post #143
andsoitis wrote: post #121

Earlier quoted context omitted.

I don’t know whether it was ever effective strategy for candidates, but I will simply say that as a hiring manager for over 12 years, I have never been interested in anyone’s resume when I see that.

Most HR departments have been filtering resumes (or LinkedIn) based on things like keywords for years before they got to you. So your reaction to resumes that heavily use those may be reactionary to being presented with tons of those (by whoever filtered them before you)

I once had an application rejected less than five minutes after submission. Hard to imagine that was anything but lacking a certain threshold keyword.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#175

Post by luotuoshangdui »

charliebwrites wrote: post #15

Anecdata, sample size of one: When I was looking for my next role after being laid off, I didn’t get much of a response with my human handmade resume despite my experience Just for kicks, I asked ChatGPT to “Analyze my resume and give it a score for what percentage it was in” then I asked it to revise it to make it score as high as possible I still tweaked and fact checked it but after I started sending that out, I g…

It makes sense. An LLM can definitely help polish your resume.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#176

Post by sxg »

1attice wrote: post #120
sxg wrote: post #64

Earlier quoted context omitted.

Take a look at how things worked before (and still do): employers decide who get jobs based on a combination of personal biases, nepotism, and ulterior motives while applicants present distorted versions of themselves and network/pull strings to put the odds in their favor. That seems more problematic.

You would be surprised at the process in other industries. What you are describing is the tech job market specifically. Other fields have their own problems, including credentialism and ballooning concomitant student loans, but do, by strict convention, not hire based on vibes or pulled strings. Often to their partial detriment, as the cure -- ie, strict oversight of hiring that also forces the hiring manager to igno…

I'm a physician and have recently been on both sides of the hiring process for new physicians and residents at a few different institutions. It's absolutely not meritocratic--you'd be shocked at how strong a role connections and pedigree play. The hard requirements are just table stakes, but the selection process from there is completely subjective and susceptible to all kinds of problematic biases. Generally people don't want to rock the boat and discuss this stuff openly, but it's absolutely a problem that needs to be pointed out.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#178

Post by dawnerd »

drillsteps5 wrote: post #77
dawnerd wrote: post #57

Earlier quoted context omitted.

I know if I got a resume from someone that had obviously used AI to generate it, it would be a pass.

Before the resume ends up in the hiring manager's inbox it needs to be picked by the recruiter from literally hundreds of others. The recruiter uses HR software to determine the match (usually the percentage), and then picks top 5% or top 20 or whatever highest ranked resumes. Guess what's doing the ranking.

That’s assuming everyone is doing it that way.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#179

Post by hyperpape »

delusional wrote: post #171
lunchbucket wrote: post #163

Earlier quoted context omitted.

Now that they've confirmed that was in fact what they meant, how have your views on this exchange changed?

> how have your views on this exchange changed? Not at all, because I am critiquing the authors writings, and for those I don't need to speculate on his intentions. He wrote a comment where he misrepresents the arguments in the paper, while explicitly saying he didn't bother to read it. That's not good enough. The author of said comment now comes in, after getting criticized, and claims that "yes, I meant that all al…

I very specifically said I read 25 pages of it in the first post of this thread. I didn’t go back, I haven’t looked at the paper since yesterday.

I read their methods and their explanation and judged them to be lacking.

The fact is, they did not measure that LLMs prefer LLM authored resumes, but that is what their paper stated.

They measured that LLMs prefer LLM authored executive summaries, which is a weaker claim.

Re: AI Self-preferencing in Algorithmic Hiring: Empirical Evidence and Insights

#180

Post by delusional »

hyperpape wrote: post #179
delusional wrote: post #171

Earlier quoted context omitted.

> how have your views on this exchange changed? Not at all, because I am critiquing the authors writings, and for those I don't need to speculate on his intentions. He wrote a comment where he misrepresents the arguments in the paper, while explicitly saying he didn't bother to read it. That's not good enough. The author of said comment now comes in, after getting criticized, and claims that "yes, I meant that all al…

I very specifically said I read 25 pages of it in the first post of this thread. I didn’t go back, I haven’t looked at the paper since yesterday. I read their methods and their explanation and judged them to be lacking. The fact is, they did not measure that LLMs prefer LLM authored resumes, but that is what their paper stated. They measured that LLMs prefer LLM authored executive summaries, which is a weaker claim.

The references start at page 28, and then the rest is appendices. If you'd read those last 3 pages you could say you'd read it all, and then maybe you could have an opinion about it.

You have to separate those two issues though. You spew out an opinion about a paper you haven't read. That's bad no matter what your opinion is. Don't blast your opinion out into the world if you haven't bothered to actually think about it first. That's one issue. A second issue is then that I think your opinion, that you didn't read the paper to make up, is wrong. They have in fact provided a justification, you just don't feel like it counts. I decided to join those two, because not reading the justification would explain why you didn't believe they had one, but that coupling isn't necessary.

They did still provide a justification, you said they don't. That's wrong. Now you're saying that you don't find it convincing, that's perfectly OK, but you then extend that claim into an accusation of misreporting. That's where you go off the rails again. They are accurately reporting what they have observed and concluded. They have provided justification for that conclusion, and all of that is, to my eyes, reported accurately.

The reason you can, accurately and correctly, claim

> They measured that LLMs prefer LLM authored executive summaries

Is exactly because their paper accurately states what they are measuring and why they believe the conclusion extends to a more general claim. You treat it as though it's some bombshell discovery, but they tell you, right in the fucking text.

If I had to revise me opinion, I guess I'd say I now no longer believe you didn't read the paper, but instead that you don't know HOW to read scientific papers.

Post reply on HN