Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

danunparsed.com

Post reply on HN
Print view View original

91–100 of 463 posts

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#91

Post by recursivecaveat »

jerrythegerbil wrote: post #4

> I fail 65% of the time. Same exact resume, different luck. As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true. 35% chance of elevating a technical individual to the next stage with no effort? I’ve seen as many as 100+ applicants an hour even when including a domain specific screener question. That’s 35 “s…

If you have no requirements for accuracy, you can just advance 35% of applicants at random.

If the first 50 people who apply are all bots, why are you reading resumes in order of submission?

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#92

Post by srdjanr »

pu_pe wrote: post #79

He tried with a tiny model (gemma3:4b), got a range from 66 to 99. Then tried again with a small model (gemini 3.1 flash lite), the range was 48 to 64. Would a frontier model be more consistent? Perhaps this tool was optimized for more capable models?

It makes sense to me intuitively (though I'm not sure if my reasoning is actually correct).

Worse model may not "know" enough to distinguish between a 70 and a 100 candidate, so it's expected that it's output has high variance. But a better model might "know" enough, so it can be more confident and thus more consistent.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#93

Post by jdw64 »

It seems like the design is flawed, probably because the scoring structure and conditions are wrong. And originally, due to the nature of LLMs, even if the input is unstructured, when you design something like a RAG system, you usually need to create a verifiable evidence table. Even with that, the scores are still probabilistic by nature, but at least they stay within an error distribution that I can verify. But it doesn't seem like there's any such evaluation criteria here.

Typically, retrieval should be tied to evaluation metrics, evidence should be linked to scores, and you also need to account for parsing errors.

But personally, I'm weak to these kinds of ATS systems (ugly appearance, non-native English speaker, didn't go to a good university), so if this kind of filtering existed, I probably would have never had a job in my entire life. Come to think of it, even now I don't have a proper job—I just bid on projects at the lowest price and implement them. So maybe it doesn't really matter whether such a system exists or not

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#94

Post by microtonal »

lelandbatey wrote: post #89
miki123211 wrote: post #64

Earlier quoted context omitted.

In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample…

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware). But theoretically, the output of every LLM is seed-driven (or could be i…

Stable seeding is not enough. A lot of modern, fast compute kernels are nondeterministic. Floating point multiplication/addition is not strictly associative and e.g. reductions can combine results from different threads in different orders (e.g. through atomic ops). You can write kernels to be deterministic, but it is generally less efficient.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#95

Post by srdjanr »

davidpapermill wrote: post #69

A better way to reformulate this problem is for the LLM to be tasked with making a _comparative_ judgement between two CVs. This should prove much more reliable, especially if you give it a third “too close to call” option. You can also ask for clear justifications of preference.

That's a good idea.

The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#96

Post by rvba »

PufPufPuf wrote: post #48
jerrythegerbil wrote: post #4

> I fail 65% of the time. Same exact resume, different luck. As someone who’s run hiring pipelines for technical roles in the past few years, that’s actually a fantastic number. I objectively hate saying that, but it’s true. 35% chance of elevating a technical individual to the next stage with no effort? I’ve seen as many as 100+ applicants an hour even when including a domain specific screener question. That’s 35 “s…

In that case, I have a pre-screening system to sell you. Through state of the art technology, it only lets through the best* 1% of applications. *According to our proprietary, undisclosed, non-deterministic metric, which may or may not be Math.random

Reminds me of this

https://stackoverflow.com/questions/16833100/why-does-the-mo...

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#97

Post by agnosticmantis »

ryukoposting wrote: post #12

At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.

A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#98

Post by Traubenfuchs »

This actually makes a lot of sense, it's testing the luck of the candidate through the rng feeding the LLM. You wouldn't want to hire unlucky employees after all! Hiring managers of the past would solve this by throwing every second resume in the trash, now this is a built in feature of ATS.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#99

Post by kailpa1 »

From `resume_evaluation_system_message.jinja`

> *SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:*

> - College, university, or educational institution name

> - CGPA, GPA, or academic grades

I don't understand why they would omit these factors from the evaluation.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#100

Post by srdjanr »

neya wrote: post #25

I wonder how is this even legal? The only useful job the HR departments are ever required to do - they decide to automate it? Aside from being a daycare for adults, what exactly does HR accomplish? It's clearly NOT on the side of employees, but this seems like they're clearly NOT on the side of employers, either. While resume's are being filtered left and right, they just make TikTok's on company's dime [1]. What a s…

They could be using this just to throw out the obviously bad CVs, and then manually go over the rest. I'm not sure if they do this in practice, but the tech itself can be useful.

Also if HR was really useless (or actively hurting the company) they wouldn't still have a job (or they'll lose it eventually). No one likes burning money for no reason. So obviously they are doing something useful.

Post reply on HN