Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

danunparsed.com

Post reply on HN
Print view View original

101–110 of 463 posts

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#101

Post by rvz »

I see.

> LLM is called six times to extract structured information

Followed by

> The default model is gemma3:4b, running at temperature 0.1 — low, supposedly nudging the model toward deterministic outputs.

This is exactly why hiring is even more broken: Because the people looking for candidates are also just as unqualified if not, more.

Using much weaker LLMs to replace the person in charge of the final judgement call is the wrong solution as this is a plain old social problem.

Even if you wanted to use LLMs for this case, the default configuration, model choice is laughably flawed. This LLM can’t be trusted as it doesn’t even know what it is reading.

The correct solution is either advanced OCR with keyword ranking with a basic filter or a far stronger LLM that excels at document / vision parsing benchmarks with an experienced person making the final judgement call in case the technology misses a critical detail.

Rather than using this less accurate one that hallucinates out its decision depending on a dice roll.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#103

Post by bhanu786 »

dvt wrote: post #6

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…

Agree

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#104

Post by sph »

kailpa1 wrote: post #99

From `resume_evaluation_system_message.jinja` > *SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:* > - College, university, or educational institution name > - CGPA, GPA, or academic grades I don't understand why they would omit these factors from the evaluation.

Hopefully so that people like me, that dropped out of high school yet have had a successful career as a self-taught engineer, have a chance. [1]

Just kidding, my resumes are sent to /dev/null like everybody else’s.

——

1: In fact, I will be controversial and say that self-taught engineers tend to be the strongest in their own particular niche, because they are powered by sheer desire to learn and improve. I am routinely appalled by how many people go on forums to ask how to learn a new thing, completely unable to self-direct their learning. I blame the modern school system.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#105

Post by jorisw »

0xpgm wrote: post #63

With such kind of ATS systems, is it still a thing to optimize for a one page resume that is easy for a human reviewer to scan, or just include enough buzzwords and external links to try and please the LLM?

I wouldn't assume based on this one thread/article that this is what you need to optimize your resume for. Nor that a majority or even significant group of reviewers is even using LLMs. I've been involved in hiring pipelines and never even thought of using LLMs to review incoming candidates.

However given the time constraints reviewers have, yes, the former (making a resume easy to consume quickly) is a huge help.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#106

Post by gregates »

psalaun wrote: post #22
rkuska wrote: post #16

This reminds me of my former CTO. He would take bunch of CVs and randomly throw some of them in a bin. He didn’t want to work with “unlucky” people.

I thought this was only an old urban legend; some people actually use this technique? Especially in a trade supposed to be led by people trained in sciences?

Given how often it's been mentioned here, it's likely that this is an urban legend that people are pretending to have first-hand knowledge of for karma. In a trade that's supposed to be led by people trained in sciences, no less!

(A more charitable interpretation would be that aforementioned CTO was making a joke that didn't land.)

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#108

Post by [deleted] »

dvt wrote: post #6

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…

[deleted]

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#109

Post by skribb »

srdjanr wrote: post #95
davidpapermill wrote: post #69

A better way to reformulate this problem is for the LLM to be tasked with making a _comparative_ judgement between two CVs. This should prove much more reliable, especially if you give it a third “too close to call” option. You can also ask for clear justifications of preference.

That's a good idea. The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.

Could probably do an elo system and sample pairs. E.g.

1. Set the elo of all CVs to 1000 elo

2. Randomly pair up CVs and compare. Winners gain elo, losers lose elo.

3. Repeat #2 for a few iterations, then remove bottom X% of CVs.

4. Repeat 2-3 until the amount of remaining CVs is small enough to do an exhaustive comparison.

I don't have a mathematical proof, but I suspect that this is a decent cost-effective approximation of comparing every pair (depending on the parameters)

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#110

Post by toolslive »

lelandbatey wrote: post #89
miki123211 wrote: post #64

Earlier quoted context omitted.

In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample…

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware). But theoretically, the output of every LLM is seed-driven (or could be i…

It's probably due to the fact that it's a cloud service. You have no guarantee that your next request will go to the same machine. So even with an identical seed, and temp 0 you might get different hardware and hence different accuracy/noise in the floating point operations.
Post reply on HN