Bot or Human? Detecting ChatGPT Imposters with a Single Question • Hacker News Board

Re: Bot or Human? Detecting ChatGPT Imposters with a Single Question

#51

Post by cdumler » Sat, May 13, 2023, 8:04 PM UTC

dismantlethesun wrote: post #45

lakis wrote: post #24

Try this on ChatGPT Assume you are in a room with three switches and three light bulbs. How will you figure out which switch controls which light bulb? It gives you the answer to the "popular" puzzle, not the simple answer of flipping a switch and see which light bulb turns on. So I think that you can take popular puzzle, modify them to make the simple and see what the answer is. If it's the answer to the popular puz…

Bard gives me some nonsense about opening the door then touching the lightbulb to see if it's hot. I think it thought this was a trick question.

The original riddle was something to the effect of:

Your are on the top floor of a building. In the room are three switches, all are off. It is given that one of the switches controls the light in the basement; however, there is no way to determine from the top floor which switch is driving the light. Is it possible just by toggling the switches as you see fit to then travel down to the basement and observe without a doubt which switch controls the light?

Yes, it is, and without any form of tricky answer (ie rewiring the switch boxes, I look down the stairs, etc).

Re: Bot or Human? Detecting ChatGPT Imposters with a Single Question

#52

Post by valine » Sun, May 14, 2023, 7:51 PM UTC

iudqnolq wrote: post #26

valine wrote: post #12

Earlier quoted context omitted.

I doubt this would work reliably if the LLM system prompt instructs it to play dumb. “Answer this question as a human with 6th grade math skills” is pretty much all that’s needed to defeat your captcha.

ChatGPT 3.5 gets the wrong answer but still doesn't sound like a regular human. Saying "we'll need to remember" about something on the right wikipedia page but not actually required to answer the question is a telltale sign of GPT: > To find the 35th digit of pi, we'll need to remember that pi is an irrational number, which means it goes on forever without repeating. However, I can still give you the value of the 35t…

I don’t understand how that’s useful as a captcha. How do you evaluate if something sounds human like other than gut feeling?

Re: Bot or Human? Detecting ChatGPT Imposters with a Single Question

#53

Post by iudqnolq » Sun, May 14, 2023, 8:02 PM UTC

valine wrote: post #52

iudqnolq wrote: post #26

Earlier quoted context omitted.

ChatGPT 3.5 gets the wrong answer but still doesn't sound like a regular human. Saying "we'll need to remember" about something on the right wikipedia page but not actually required to answer the question is a telltale sign of GPT: > To find the 35th digit of pi, we'll need to remember that pi is an irrational number, which means it goes on forever without repeating. However, I can still give you the value of the 35t…

I don’t understand how that’s useful as a captcha. How do you evaluate if something sounds human like other than gut feeling?

My gut feeling is that machine learning is pretty good at this kind of text categorization. You could also try going old school and using stylometry. There are a bunch of companies trying to sell GPT detection.

But also remember captcha stands for Completely Automated Turing Test to Tell Humans and Computers Apart. Automated isn't necessarily always a hard requirement, for a Turing Test just using your gut is fine.