Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

danunparsed.com

Post reply on HN
Print view View original

111–120 of 463 posts

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#111

Post by nok22kon »

lelandbatey wrote: post #89
miki123211 wrote: post #64

Earlier quoted context omitted.

In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample…

As I understood it, the "randomness" affecting what is selected at any temperature still comes from a PRNG or CSPRNG (or whatever RNG you want, maybe a hardware one), and if you where to swap out that with something deterministic you'd get the same results every time (barring non-determinism in other parts of the OS/drivers/maybe even hardware). But theoretically, the output of every LLM is seed-driven (or could be i…

that's incorrect in the presence of batching. it's tough work making it truly deterministic:

https://x.com/FireworksAI_HQ/status/2069873437217276015

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#112

Post by DougBTX »

317070 wrote: post #47
aesthesia wrote: post #35

Earlier quoted context omitted.

A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.

> so in principle, setting temperature to 0 _should_ result in deterministic outputs It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the…

> GPUs put the associativity of the sums in matrix multiplications in arbitrary order

That’s user-controlled too, not an inherent property of GPUs:

https://docs.pytorch.org/docs/2.12/generated/torch.use_deter...

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#113

Post by nok22kon »

dvt wrote: post #6

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…

its a bad idea in general to use non-1.0 temperature. there is a reason labs are strongly recommending using 1.0.

using low temperature is more deterministic, but the cost is the model becomes "dumber"

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#114

Post by jstanley »

317070 wrote: post #47
aesthesia wrote: post #35

Earlier quoted context omitted.

A distribution with all probability mass on one outcome is deterministic, so in principle, setting temperature to 0 _should_ result in deterministic outputs. There are a few reasons it might not, but I don't think any of these apply when running a local model like the author did.

> so in principle, setting temperature to 0 _should_ result in deterministic outputs It is a common misconception, but it is not true even in principle. If I have 2 or more logits which are equal to the maximum of my logits, I will sample uniformly random from them with any temperature, even zero. Sampling from softmax([1, 0, 1]) is still stochastic at temperature 0, because the limit is to sample uniformly from the…

> "GPUs don't do deterministic matrix multiplications" is the biggest source of randomness in LLMs.

But this isn't a fundamental property of LLMs, it's just an implementation detail. It's pretty obvious that if you evaluate the matrix multiplications correctly and deterministically sample from the highest-probability outputs, you will have a deterministic LLM.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#116

Post by swiftcoder »

kailpa1 wrote: post #99

From `resume_evaluation_system_message.jinja` > *SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:* > - College, university, or educational institution name > - CGPA, GPA, or academic grades I don't understand why they would omit these factors from the evaluation.

> I don't understand why they would omit these factors from the evaluation.

Only hiring MIT graduates sounds great to a lot of tech folks! Automatically rejecting applicants from HBCUs, however, sounds like a lawsuit

As to GPA thing, I think it's just to stop the LLM glomming onto an obvious numerical grade? LLMs like to rank things by obvious dimensions, and whether someone had a 4.0 or a 3.8 in grad school makes very little difference to their performance 10 years down the line.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#117

Post by swiftcoder »

srdjanr wrote: post #95
davidpapermill wrote: post #69

A better way to reformulate this problem is for the LLM to be tasked with making a _comparative_ judgement between two CVs. This should prove much more reliable, especially if you give it a third “too close to call” option. You can also ask for clear justifications of preference.

That's a good idea. The only drawback I see is that you should compare every pair of CVs for best results, and that grows quadraticly with number of CVs. Of course you can settle for fewer comparisons and not perfect results. But then I'm not sure if you can hit a good ratio of quality and token spend.

> you should compare every pair of CVs for best results

Or compare each one to a reference set? Take 5 resumes of existing employees, rank all candidates against that set, maybe you get some useful level prediction into the bargain

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#118

Post by kailpa1 »

sph wrote: post #104
kailpa1 wrote: post #99

From `resume_evaluation_system_message.jinja` > *SCORES MUST NEVER DEPEND ON THE FOLLOWING FACTORS:* > - College, university, or educational institution name > - CGPA, GPA, or academic grades I don't understand why they would omit these factors from the evaluation.

Hopefully so that people like me, that dropped out of high school yet have had a successful career as a self-taught engineer, have a chance. [1] Just kidding, my resumes are sent to /dev/null like everybody else’s. —— 1: In fact, I will be controversial and say that self-taught engineers tend to be the strongest in their own particular niche, because they are powered by sheer desire to learn and improve. I am routine…

I'm a self-taught programmer as well, who dropped out of university, and these factors being omitted would benefit me as well, but I feel like good grades and a good university are still indicators of someone being or is capable of becoming a good programmer.

This system would drop a Harvard top graduate for someone having a year of experience in some outsourcing firm.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#119

Post by throwawaythekey »

agnosticmantis wrote: post #97
ryukoposting wrote: post #12

At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.

A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.

> A person's total luck is constant over a lifetime

Ah yes, the much revered cosmological fairness constraint.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#120

Post by padolsey »

This is just the 'LLM judge', very badly implemented without any scientific prudence. What a joke. To be terse: you cannot rely on LLMs to provide standardized scores against arbitrary criteria. To get close to 'reliable' you would need highly tested rubrics, grounded in human decision-making, and you'd need to avoid all the measurement biases these things are riddled with... positional/order effects, anchoring on whatever numbers you stuffed into your own prompt, scale-format sensitivity (a 1–5 and an A–E scale give different answers for the same input), holistic-vs-isolated context effects, and lovely examples like where adding a "be unbiased" instruction makes it more biased. I've studied this at length. You cannot even _begin_ to approach this problem seriously without held-out validation, inter-rater agreement, and ground truth. This repo is just quagmire of wishful vibes with random numbers littered throughout.
Post reply on HN