In my experience this complete lack of reproducibility is what happens when you throw LLMs at a complicate problem without sufficient shaping, workflows, etc. Go ahead, let LLMs invoke LLMs invoke LLMs and by the end you'll get an output that's really well written but completely different run to run.
HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
411–420 of 463 posts
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#412> I’d take the engineer with 30 years of experience who built S3 over someone with two internships and an open source project — but this tool wouldn’t. Is it possible the senior/principle jobs are not being applied to at a rate that LLM tools like this are required? Maybe star devs are getting recruiter referrals and this kind of tool is mostly used for filtering new grads? Either way, perfectly dystopian.
That's what stood out to me as well, and it struck me as odd that nobody seemed to think it's odd? Almost half of the points to be made is related to contributions to open source projects. Guess my 10+ years of experience in a niche topic is worthless.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#413In my experience this complete lack of reproducibility is what happens when you throw LLMs at a complicate problem without sufficient shaping, workflows, etc. Go ahead, let LLMs invoke LLMs invoke LLMs and by the end you'll get an output that's really well written but completely different run to run.
If those actually solved the fundamental issue we would already have an explosion of competitive software for major projects
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#414Earlier quoted context omitted.
That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal. That's why greedy sampling generally produces deterministic output for LLMs. The real gotchas are elsewhere (like with batch inference as we've seen with earlier GPTs). But unlike what the earlier comment says, this is a non-issue mathematically.
> That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal Any two tokens ending up with the exact same logit is very unlikely, but not impossible; and as the number of output tokens grows, the odds that it will happen eventually gets higher and higher. I suppose, to ensure determinism, rank by logit then token ID, so…
You aren't looking for a random set of tokens that have the exact same logit, you are looking for the largest n tokens to have the exact same probability.
This is exceedingly unlikely, as training will only push one of them up for any individual sample. There are likely some pathological situations that could end up with that situation, maybe, but it is pretty unlikely in a general case.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#415Earlier quoted context omitted.
Every time people point out a limitation or constraint of LLMs, I see a comment that is to the effect of “but humans…”. I don’t understand why this comparison is relevant to this particular thread. Is it just an amusing similarity?
It may seem trite but the point is that if separate humans were assigned the same task the LLM was here the results would be similarly non-deterministic.
Indeed: LLMs do tasks that would otherwise be assigned to humans. So when pointing out deficiencies in LLM performance they should be compared to the alternative, which also isn't perfect.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#416At this point we might as well adopt that joke where you blindly throw away half the resumes because you don't want to hire unlucky people.
A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.
Normally we'd reject the first 37% [0] of candidates and then pick the next one that is above the average, but if all the unluckiest candidates show up first, then we need to sample even more in order to get an accurate baseline.
This may be compounded by the the "Teela Brown" problem [1], where some candidates may be too lucky to end up with our company, causing them to appears later in the stream or not-at-all.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#417The blog post itself has pretty a pretty strong un-copy-edited ChatGPT vibes.
Yeah, this type of thing makes me sad. It's a good idea and the work behind it is interesting, but there's something magic about a human voice. It deadens me a little bit reading this type of writing, which I'm seeing increasingly in both my work and personal life.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#418Earlier quoted context omitted.
So the logical solution is for candidates to submit multiple applications with slight variations to their contact info, "John Schmidt", "John J. Schmidt", "John J. J. Schmidt", "John Jacob J. Schmidt", "J. J. Jingleheimer Schmidt", etc.
Hey, that's my name too!
Whenever I send them out
The filters always route:
"Spammer: John Jacob Jingleheimer Schmidt"
[N/A] [N/A] [N/A] [N/A]Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#419so if a scoring system automatically ranks candidates lower for lacking a public GitHub or open-source contributions, that might indirectly disadvantage certain groups. ie
ppl in defense, security, or other restricted industries who cannot legally share work publicly, people under strict employer IP/confidentiality rules, some demographics or nationalities with different access to open-source engagement (something like non-citizens not allowed to use xyz ie mythos), this can be construed as disparate impact which is protected under the equal employment opportunities commission (EEOC) and employers can be held liable for this as it can be seen as a biased discriminatory practice.
Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88
#420An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…
In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample…
I did large scale tests temp 0 and there was still randomness with the same prompt inputs coming in.
I did this with several model apis.
GPU processing is not going to be the same from what I read but also the AI backend is doing a lot of fancy batching resulting in another layer of randomness.