Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

danunparsed.com

Post reply on HN
Print view View original

291–300 of 463 posts

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#291

Post by Grombobulous »

bob001 wrote: post #173
CM30 wrote: post #154

I think what's more worrying to me (if other systems work like this ATS) is that it seems to judge based on a bunch of factors that will probably disqualify a ton of decent to good participants. For example, 65 points are given for a mix of personal projects and open source contributions. Which is great if your one and only interest is in tech, and you don't have a family, dependents or a second/third job. If you hav…

[flagged]

“Fair” is one thing, “systemically impossible to even approach fair” is another.

For example, you can’t “conscious long-term effort” your way out of being stop and frisked by cops because you were walking while black.

This setup isn’t even good for employers. Having your job as your hobby doesn’t automatically make you better at your job.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#292

Post by rkozik1989 »

StilesCrisis wrote: post #270
sigmoid10 wrote: post #196

Earlier quoted context omitted.

That is not a problem for LLMs, because in practice floating point inaccuracies (in particular after exponentiation) prevent values from being exactly equal. That's why greedy sampling generally produces deterministic output for LLMs. The real gotchas are elsewhere (like with batch inference as we've seen with earlier GPTs). But unlike what the earlier comment says, this is a non-issue mathematically.

"Makes unlikely" is very different from "prevents." If there's one counterexample, it's not really deterministic.

Exactly, consider the scenario where laws are at play and violating them could cost companies thousands. Recently my father received a 'request for address' letter addressed to me at his nursing home, the building has always been a nursing home, and he's also in his mid-70s. That's very obviously a violation of the Fair Debt Collection Practices Act. Imagine the implication of this if the law firm in questions used an AI-assisted data enriching product to find this information. That SaaS company is not only liable to that one law firm but every law firm who uses their software. Its potentially a federal class action lawsuit.

My point is, deterministic logic matters in certain circumstances 100% of the time. Forcing the LLM to make something unlikely is not good enough because a series of mistakes could very quickly bankrupt the company.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#293

Post by joshmn »

I ran the ATS myself and had a similarly quirky experience. I was in the 70s because it couldn't find my GitHub profile, and then it didn't like some of the popular Ruby libraries I'm the author of.

After a few runs it picked things up appropriately. I always got dinged on formal education though.

This stuff is gross.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#294

Post by bluGill »

buzer wrote: post #277
dathinab wrote: post #247

And this + the tendency for AI to "prefer" AI produced code + some other AI biased is why *this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways. To be clear: - randomly filtering "too many" resumes is pretty much allowed (I think) - but must be actual random independent of the resume (and can be in multiple layers, i.e. random filter > pre-select > random filt…

> this is most likely highly illegal to use in the EU due to violating anti discrimination laws in multiple ways. It's generally illegal under GDPR Article 22. > The data subject shall have the right not to be subject to a decision based solely on automated processing, including profiling, which produces legal effects concerning him or her or similarly significantly affects him or her. Exceptions in 22(2) are unlikel…

For C: I'm not sure how EU laws work, but ethics says that someone who needs a job cannot give consent since the possibility of a job if they give consent could be a bribe. See a lawyer for how it works in your country.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#295

Post by vessenes »

Dylan16807 wrote: post #204
vessenes wrote: post #200

Earlier quoted context omitted.

To be clear, temperature 0 is deterministic and will produce the same output for exact duplicate inputs, across all seed choices. Provided: * If it’s MoE we are talking about, that the duplicate inputs are for the whole batch (yes, your batch neighbours can impact your choice of experts. Blergh.) * Your kernels are deterministic * There’s no system wide effort switch that responds to, e.g. work load across the cluste…

Even then it's deterministic in the way a hash function is deterministic. Change one letter and you can get a completely different output. What people actually want is something continuous.

Agreed on the desire for continuous behavior. That said, in a modern LLM, is this hash analogy accurate? I would be surprised if a single letter changed most zero temp force ranked outputs.

E.g:

“Where is the Eiffel Tower Located? One word only.”

“Where is the Effel Tower located? One word only.”

“Where is the Eiffel Tower located? One wor only.”

I’d be very surprised if those got different answers from even a small local model at temp 0.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#296

Post by freehorse »

StilesCrisis wrote: post #275
sigmoid10 wrote: post #181

Earlier quoted context omitted.

The point is that the case T=0 doesn't just "exist" as a special code branch - it is still well defined mathematically without any change to the output function. What the above comment refers to with the extra "if" check is just a limitation of computers not liking to divide anything by zero, even if the actual function exists and is well behaved at zero. It is not some weird or special theoretical construction.

Floating point defines n/0 the same as math. It's infinity as long as n isn't zero.

> as long as n isn't zero

Which is the case with softmax function, as for T=0 you end up with a fraction that either becomes 0/0 or inf/inf [0]. So you do need branching as floating point arithmetic is not gonna get you there.

[0] except for weights that are exactly 0

edit: thinking more about it, one could always express the softmax formula in ways that this could work with floating point arithmetic but it would be very inefficient and sort of pointless

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#297

Post by simiones »

mtharrison wrote: post #280
dvt wrote: post #6

An alarming number of people don't understand that LLMs work via purely stochastic processes, so I'm happy to see in-depth pieces like this. I'm looking for a job and maybe this is why it's so hard to get a callback these days: resumes are just dumped in some LLM black hole and no one really knows how it works. The author says: > temperature 0.1 — low, supposedly nudging the model toward deterministic outputs This is…

Small refinement: the underlying model isn’t stochastic at all. The forward pass is a deterministic function of the weights and input, it just produces a probability distribution over the next token. The stochasticity is an optional sampling step layered on top, not something inherent to LLMs. Greedy/argmax decoding (or temperature 0) makes the whole thing deterministic. So “purely stochastic” overstates it a bit: th…

There are more layers to this problem, if we want to get into the details. The LLM is defined in terms of floating point operations, and those are not actually fully deterministic, on most hardware and in most performant implementations.

IEEE 754 only specifies precision requirements for certain operations, not precise bit patterns (e.g. for exponentials). So, at least in principle, the same hardware performing the same operation could produce different results at different times, as long as they are close enough to the theoretical answer. I'm not sure if any hardware actually works like this.

IEEE 754 also specifies that many of the basic arithmetic operations are not associative - so any reordering (which is common when batching multiple queries at the same time) will introduce indeterminacy from the perspective of your own query (that is the result for your query will change depending on what other query happens to be processed at the same time, which is not under your control).

Finally, even if we take the case when a query is processed alone, and even if one particular hardware is completely deterministic, the result will be different on different hardware - which can again look like non-determinism if you're sending your query to a load balancer.

So, the math for LLMs is deterministic in theory, but implemented with non-deterministic approximations & optimizations in practice, and their results are then normally used only as a probability distribution to be sampled from.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#298

Post by efromvt »

mahogany wrote: post #218
spwa4 wrote: post #58

Earlier quoted context omitted.

[flagged]

Every time people point out a limitation or constraint of LLMs, I see a comment that is to the effect of “but humans…”. I don’t understand why this comparison is relevant to this particular thread. Is it just an amusing similarity?

I think it often useful to push the conversation down "we built a system for humans that dealt with this, what from that is or is not applicable for agents in the same context"? Humans randomizing resume review for screening is pretty known; I've seen companies try to fight it with things like hiding information, panel reviews, etc - it's unclear to me how effective those would be for agents (honestly, it was unclear how effective those were for humans). I was depressed about the hiring process before we had AI screening and I remain depressed about it.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#299

Post by Xirdus »

Arodex wrote: post #243
RugnirViking wrote: post #222

Earlier quoted context omitted.

great question. The alternative is not accepting 1000 applicants. Nobody said you have to keep up your job posting for two weeks, or two hours for that matter. stop once you have enough. Enough is defined by whatever number you would have filtered to. In the rare case none of the first ten applicants were appropriate, just open it again until youve got another tranche.

That's just another type of randomness (who was online during the short time the posting was opened).

"Being online during the short time" heavily favors bots. In a way, AI screening tools saved us from the future of everybody buying resume-spamming-as-a-service because it became as important to use these as getting a college degree.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#300

Post by vidarh »

chrisjj wrote: post #156
vidarh wrote: post #147

Earlier quoted context omitted.

And this lies at the heart of the problem. We expect computers to be consistent despite running programs that are not designed to be consistent. This despite the fact that we have lots of experience of programs running on computers that produces wildly inconsistent outputs. But for some reason some people choose to assume LLMs should act like a calculator instead of any of those programs.

> This despite the fact that we have lots of experience of programs running on computers that produces wildly inconsistent outputs. The average user has very little. A word processor with inconsistent pagination or a spreadsheet with inconsistent totals is rightly seen as faulty.

The average user is familiar with games.
Post reply on HN