Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

danunparsed.com

Post reply on HN
Print view View original

431–440 of 463 posts

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#431

Post by fireant »

CM30 wrote: post #154

I think what's more worrying to me (if other systems work like this ATS) is that it seems to judge based on a bunch of factors that will probably disqualify a ton of decent to good participants. For example, 65 points are given for a mix of personal projects and open source contributions. Which is great if your one and only interest is in tech, and you don't have a family, dependents or a second/third job. If you hav…

In my experience personal projects are the greatest indicator of IC competence, especially for young people. You may not like it, but turns out that when you do a thing in your free time because you like it, you get better at the thing than the people that only do it because they have to.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#432

Post by YossarianFrPrez »

WickyNilliams wrote: post #429
YossarianFrPrez wrote: post #134

Looking at the linked scoring prompt (resume_evaluation_criteria.jinja) [0], I immediately see several red flags that suggest the output won't be reliable. (I'm developing an LLM intensive application where the stakes are high enough that I need the LLM output to be reasonably correct.) [0] https://github.com/interviewstreet/hiring-agent/blob/main/pr ... In no particular order: 1. The prompt is trying to get the syst…

Would it be correct to say you have experience building LLM based workflows like this? I'm guessing so, given by your critiques and suggestions of better approaches. Can you recommend any books/sites/other for learning these kind of dos and donts?

While I have some experience, I'm afraid I am learning mostly through a) trial and error, and b) using an llm chat interface to double check my thinking.

This being said, here are my top recommendations:

1. Build your system against real targets. Had HackerRank continually tested their prompt against 2-3 real resumes that were scored by hand, I think some of the issues would have immediately popped out. The people who built the prompt thought they could magically skip the hard part of articulating a preferred decision making process by having the LLM do it. But LLMs are much better at scaling a pre-existing decision-making process rather than inventing one, let alone the same one, from scratch, every time.

2. Think about what it would take to get motivated undergraduate interns to do the task from end to end, step by step. That's essentially what your workflow will need.

3. If the LLM can't do a step or sub-task reliably, then it's time to decompose those subtasks into even smaller chunks.

I'm sorry I can't be more helpful!

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#433

Post by tptacek »

tikhonj wrote: post #421
nonethewiser wrote: post #379

Earlier quoted context omitted.

>Which means there's a good chance this is somehow correlated in one way or another to race/gender/other protected classes in the US, just by the math of everything being correlated to everything. >Which means this is one good lawsuit away from being illegal in the US as well. Uhh.. what? No that doesn't follow at all. Screening resumes in a way that correlates to race, gender, etc. is not illegal. This is a fundamen…

The US has a notion of "disparate impact"[1] that means you can be liable for discriminating based on a protected characteristic on the basis of correlation. This is why HR departments are very hesitant to use things like IQ tests for screening candidates, for example. [1]: https://www.congress.gov/crs-product/IF13057

If they are, they're falling for an urban myth, because IQ tests in white-collar employment are legally fine, and several household-name corporations use them openly. I'm a noodge about this particular issue because there's a folk belief that IQ tests are the secret perfect hiring mechanism; it wouldn't bother me so much if it was so obvious how badly a pure IQ-test process would perform for a job in our field.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#434

Post by miki123211 »

sigmoid10 wrote: post #72
miki123211 wrote: post #64

Earlier quoted context omitted.

In theory, temperature 0 does make the LLM deterministic. Well, in theory theory, temperature 0 doesn't really exist. Mathematically, as lim temperature->0, the distribution gets spikier and spikier, the most likely sample goes to almost-but-not-quite infinity and the rest go to almost-but-not-quite 0. In practice, temperature=0 is literally a separate branch of an if statement that just picks the most common sample…

>in theory theory, temperature 0 doesn't really exist. It does exist very much, even if you go to pure math. Look at the softmax function and take the limit as T->0. It becomes a dirac-delta function. I.e. in a discrete setting (like for LLMs with a finite set of output tokens), probability P becomes one for argmax and 0 for everything else. Only in coding practice it is easer to implement T=0 as a simple if check th…

There's a difference between f(0) and lim t->0 f(t).

We just chose to treat this function as a "staircase function" where f(0) =lim t->0 f(t), general formula for f(t!=0).

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#435

Post by kailpa1 »

Schiendelman wrote: post #245
kailpa1 wrote: post #118

Earlier quoted context omitted.

I'm a self-taught programmer as well, who dropped out of university, and these factors being omitted would benefit me as well, but I feel like good grades and a good university are still indicators of someone being or is capable of becoming a good programmer. This system would drop a Harvard top graduate for someone having a year of experience in some outsourcing firm.

Unfortunately, graduating from Harvard is a very good predictor of whether your parents were wealthy, and also that you are less likely to be black. I worked for a very large job board for the last six years, it's the one you're thinking of. What we found is that the outcomes of paying attention to what school you went to are almost entirely discriminatory, and not predictors of success.

That's sad to hear. I guess I'm giving too much credit to schools as a measure of success.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#436

Post by aspenmayer »

sfn42 wrote: post #360
agnosticmantis wrote: post #97

Earlier quoted context omitted.

A person's total luck is constant over a lifetime. The remaining half of the candidates already spent some of their luck in this selection, so they'll be on average less lucky than the discarded half.

This is not at all how probability works. Luck is not a resource one spends. If you flip heads 500 times in a row with a fair coin, the next coin flip is still 50/50.

Presupposing that the same coin is used for every flip (which is implicit in the example), it would be fair to question whether the coin could possibly be a fair coin after 500 heads in a row, even (and especially) if the flipping process were ideally fair.

I’m not a whiz with the math involved, but I am of the opinion that 500 consecutive same-side flips is a large enough sample size to calculate that the coin in question is biased, so it would be unreasonable to assume that the next flip is 50/50.

https://en.wikipedia.org/wiki/Checking_whether_a_coin_is_fai...

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#437

Post by AlexeyBelov »

BeetleB wrote: post #406
Xirdus wrote: post #305

Earlier quoted context omitted.

> If you have 1000 applications for every job, and you know that a bunch of these applications are "a bad fit", to put it mildly, you have to filter. And you cannot realistically give every resume a good, human look. At 10 seconds per resume, it would take you 3 hours to go through all 1000 resumes. I don't know what you consider "good" and "human", but my human eyes could easily do good enough, fully manual pre-scre…

> At 10 seconds per resume, it would take you 3 hours to go through all 1000 resumes. At 10 seconds per resume, I would not assume that you're screening better than the LLM.

You could remove the least relevant resumes very quickly. Maybe not 10 seconds, but 30 seconds per resume for sure.

What I hear happens now: people apply for a "Senior Golang SWE" with 2 years of experience with C#. Or, relevant for hiring in the US, the job posting says the visa is required, but people apply without it anyway.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#438

Post by Mumps »

Bedon292 wrote: post #332
a3w wrote: post #321

What does ATS mean? Neither github repo nor article explain that.

Probably: Applicant Tracking System. Used for tracking the people who apply to each of your openings, and the hiring workflow. Where this would likely be used to neck down all the applicants before a person actually looks at them to make judgement calls on who to move forward in the process.

Thank you!

Christ in pijamas. TLAs should be a capitol offence. Even worse so, somehow, when undefined.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#439

Post by sfn42 »

aspenmayer wrote: post #436
sfn42 wrote: post #360

Earlier quoted context omitted.

This is not at all how probability works. Luck is not a resource one spends. If you flip heads 500 times in a row with a fair coin, the next coin flip is still 50/50.

Presupposing that the same coin is used for every flip (which is implicit in the example), it would be fair to question whether the coin could possibly be a fair coin after 500 heads in a row, even (and especially) if the flipping process were ideally fair. I’m not a whiz with the math involved, but I am of the opinion that 500 consecutive same-side flips is a large enough sample size to calculate that the coin in qu…

I already said the coin is known to be fair.

Re: HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88

#440

Post by aspenmayer »

sfn42 wrote: post #439
aspenmayer wrote: post #436

Earlier quoted context omitted.

Presupposing that the same coin is used for every flip (which is implicit in the example), it would be fair to question whether the coin could possibly be a fair coin after 500 heads in a row, even (and especially) if the flipping process were ideally fair. I’m not a whiz with the math involved, but I am of the opinion that 500 consecutive same-side flips is a large enough sample size to calculate that the coin in qu…

I already said the coin is known to be fair.

> I already said the coin is known to be fair.

The coin can be assumed to be fair before the flips, but after the flips we have no mathematical reason to believe that the coin is fair, as our results heavily suggest that it quite simply isn’t fair.

However, for the purposes of discussion, assuming an ideal coin, the probability of 500 same flips in a row is so statistically unlikely that fairness of the flipping process and/or flipper must then be called into question.

Even if the coin, flipping process, and flipper are all ideal (which wasn’t stated, but we will also assume for the sake of argument that we have ideal immovable goalposts), the likelihood of the 500-in-row event is so improbable as to be unreasonable to use as an example or even a metaphor, because it doesn’t have much predictive power in a conversation, as ideal coins don’t exist any more than ideal coin flippers.

Even a coin designed to be fair would be fairly deemed defective after so many same-side flips, and any reasonable gambler would demand that the coin be replaced with another; at that point an argument could reasonably be made to also change the coin flipper and/or the venue.

Post reply on HN