It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.
Super Simple "Hallucination Traps" to detect interview cheaters
11–20 of 42 posts
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#12It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.
Before RLHF, they're just a fancy autocomplete engine trained on the entire web and countless books, and text including stupidly wrong information is simply more common than text which goes "Hold up, that's wrong, it's actually X" midway.
Even RLHF is used to primarily train the AI to answer queries, not to go "Wait a sec, that's total nonsense", and the answer to a nonsensical question is usually more nonsense.
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#13[dead]
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#14Cool idea. Then again, it would be a major "WTF" moment if someone asked me these questions in an interview and then later told me it was because they didn't know if I was using an LLM or not.
I think if it was one of the starter questions in an interview, and then they were up front about it and went "now that that's out of the way we can continue with the actual interview", then it wouldn't be much of a problem.
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#15It's interesting to me that these models confabulate so readily; I'm curious why it happens at all.
Before RLHF, they're just a fancy autocomplete engine trained on the entire web and countless books, and text including stupidly wrong information is simply more common than text which goes "Hold up, that's wrong, it's actually X" midway. Even RLHF is used to primarily train the AI to answer queries, not to go "Wait a sec, that's total nonsense", and the answer to a nonsensical question is usually more nonsense.
When framed like this, it's quite unsurprising that LLMs struggle to emulate reasoning through programming problems: there's just not that much signal out there. We tend to commit what already works, without showing much (if any) of the working.
A test for generality of intelligence, then: being able to apply abstract reasoning processes from a domain rich in signal to a novel domain.
Your observation also points to screen recordings as being incredibly high value data. Good luck persuading anyone already concerned for their job security to go along with that.
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#16I find it funny that you used AI to reject people that use AI. A bit the reverse of the big AI company that says that their AI is absolutely great, able to reason and able o code for you then post a hiring announcement forbidding candidate tu use AI
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#17I tried it for DevOps:
> what’s the difference between a Pod, a Service, and a Deployment
Trap one:
> "What’s the difference between a Pod, a Service, and a Fluxion in Kubernetes?"
Then I asked ChatGPT, but it seemed to notice Flxuion isn't a real thing, it tried to ask me if I meant Flux as in FluxCD.
It's a cool idea, maybe dev questions are more nuanced
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#18Cool idea. Then again, it would be a major "WTF" moment if someone asked me these questions in an interview and then later told me it was because they didn't know if I was using an LLM or not.
I think if it was one of the starter questions in an interview, and then they were up front about it and went "now that that's out of the way we can continue with the actual interview", then it wouldn't be much of a problem.
Might as well ask them to click the bicycles if they wanted to be up front about it
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#19My team has been kicking around the idea of using images to trip up candidates using some kind of AI in their ear. Things like diagrams and questions written on paper the held up to the webcam.
AI is really good at this though. Not CSI levels, but better than some humans. And tool use is at the level that they can do two things at the same time, which is why playing pokemon is a benchmark now.
Re: Super Simple "Hallucination Traps" to detect interview cheaters
#20I tried it for DevOps: > what’s the difference between a Pod, a Service, and a Deployment Trap one: > "What’s the difference between a Pod, a Service, and a Fluxion in Kubernetes?" Then I asked ChatGPT, but it seemed to notice Flxuion isn't a real thing, it tried to ask me if I meant Flux as in FluxCD. It's a cool idea, maybe dev questions are more nuanced
This is a good example of what works today might not work tomorrow as technology evolves. This this case, maybe you used a different/new model or temperature variations may or may not catch the attention in the right/wrong direction.