Super Simple "Hallucination Traps" to detect interview cheaters

Isn't this essentially the idea in blade runner? Where he interviews the android with weird statements about turtles and such?

I'm not in academia, but I happen to find this type of question helpful, when done well.

When questions make no sense and it takes a lot of effort to find out, I would agree that this is stupid and not testing for any real skill. But when questions are designed in a way to meet the knowledge level that is expected, I think this type of questions is good.

For example:

  For what x does the value of function 1 / sin(x) become zero

This question leads you astray, but it is a genuine sign of understanding when the answer is "none". OK, this is not a real trap question, but it borders on one.

A more callous example, not a MINT question (not sure what kind of test would ask this question though):

  A hotel room costs 400$ a night, breakfast not included. It is situated in NYC and the cost of a hotel room in NYC averages at 250$ per night. The average cost for breakfast is 50$. Hotel rooms in Manhattan average 500$ per night, while hotel rooms in Queens average 120$/night. In what part of NYC is the hotel located?

The answer one gives to this question could be quite revealing. If so says "it might be in Manhattan, hotel rooms are particularly expensive there, but it is not possible to give a definite answer", fine.

If someone starts bullshitting, not so good.

Another one at high-school level maths:

  A room has one wall that is 16ft long, another one that is 24ft long. What is the area of the floor of the room?

It might be reasonable to assume a rectangular room, but it's not given. So it should be expected to give a nuanced answer.

Even more callous would be to say the room is rectangular and then point out that the floor might be tilted :D

But yeah, I would be pretty annoyed by that, too. I mean, nobody would say that it's a good answer to start fretting about curved space-time or something given this question.

But in every domain, I think it's possible to design good "trick questions".

The more I think about it, this type of question is basically the same type of question one would use to "benchmark" an LLM.

And again, I'm not saying that I'd answer these correctly...

I would ask myself what am I actually paying for here. -- as mentioned in other comments, they could always have a peer next to them during a call, so hallucinations won't do --

+ Using AI is actually cheating or being productive for the role? + Am I worried that they'll do all their job in 5 minutes and afterwards do something else?

Maybe you are worried about them not being able to actually do the job, which probably means the interview process was wrong from the start. Alternatively, the performance expectations may be higher for the role; e.g. what before was 1x now needs to be 5x productivity.

As an alternative, I've heard of many SMBs opting for a model in which the last bit of the hiring process includes some paid work for a week to see how they actually perform, or checking references in depth.

Hey, to clarify - there is no product here, you are not paying for anything. This is an idea being shared, the web app was built completely for fun (using some retro styles) in about a half hour using some cursor prompts (and reviewing at the end for security) and will never become a product in the future.

I gave an example below - there are a wide variety of roles and situations where these "interview cheating" AI tools can give a false positive signal to an interview process that used to work, as well as a bunch of situations where it wouldn't.

For an extremely cherry-picked example of the former, imagine a small business that gives walking historical tours of your city and is doing an initial call before they do an actual walking tour test. Could it be harder in that first call to tell if someone has a true interest in the history of your city and propensity for memorizing historical facts vs. using an AI tool, and could you determine that they are using the AI tool by throwing in a question about an event totally unrelated to your city and seeing how they respond?

What I hate about these questions is that they're exactly what's asked in exams, so you're expect to make assumptions or fail. At this point, LLMs etc have more critical thinking to dodge these than humans who are conditioned into doing this after a decade of schooling.

There was a little entrepreneurship workshop I went to once. The trainer put a pen on the floor, gave us a ball, and asked us to stand behind the pen and throw the ball into a box. It was to demonstrate that most people didn't practice throwing before entrepreneurship and then blamed the environment for their lack of planning. I picked up the pen and moved it right next to the box so that I could walk there and put the ball in. I thought this was the actual solution (e.g. entrepreneurs were supposed to be creative), but was "failed" for "cheating".