Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

AI21 Labs concludes largest Turing Test experiment to date

ai21.com

Post reply on HN
Print view View original

11–20 of 45 posts

Re: AI21 Labs concludes largest Turing Test experiment to date

#11

Post by danpalmer »

caddemon wrote: post #5

After playing the game they used (linked at top of article) I find it hard to draw much conclusion from this study. There is a quite short timer on not only the entire conversation, but on each response you can type. When the timer runs out it sends your message in partially written form. It seriously stifles what you can ask the other "person" and it makes responses artificially short even to a deeper question. When…

I had a bot send 1 message and then quit. I naturally assumed that bots wouldn't just quit, and 1 message isn't really enough to gauge anything, so yeah based on this experience I wouldn't draw much of a conclusion from it.

Re: AI21 Labs concludes largest Turing Test experiment to date

#12

Post by caddemon »

danpalmer wrote: post #11
caddemon wrote: post #5

After playing the game they used (linked at top of article) I find it hard to draw much conclusion from this study. There is a quite short timer on not only the entire conversation, but on each response you can type. When the timer runs out it sends your message in partially written form. It seriously stifles what you can ask the other "person" and it makes responses artificially short even to a deeper question. When…

I had a bot send 1 message and then quit. I naturally assumed that bots wouldn't just quit, and 1 message isn't really enough to gauge anything, so yeah based on this experience I wouldn't draw much of a conclusion from it.

Interesting, I had a human quit, surprised bots might quit as well.

Re: AI21 Labs concludes largest Turing Test experiment to date

#14

Post by caddemon »

dinkblam wrote: post #8
caddemon wrote: post #5

After playing the game they used (linked at top of article) I find it hard to draw much conclusion from this study. There is a quite short timer on not only the entire conversation, but on each response you can type. When the timer runs out it sends your message in partially written form. It seriously stifles what you can ask the other "person" and it makes responses artificially short even to a deeper question. When…

> There is a quite short timer on not only the entire conversation couldn't agree more and they took like 30 seconds to type a few words. if i really have been talking to a human here, i can only suspect heavy usage of drugs: https://ibb.co/CHG2VcS kinda seems like this is fake or maybe i am not aware that "elbows" are a thing you can be into now - maybe a trending new fetish?

I bet there are people purposely screwing with it, since they designed it to be 2 sided when you get paired with a human. The actual Turing test is not supposed to be this way, though it still relies on at least some assumption of good faith (or properly incentivized) participants.

Re: AI21 Labs concludes largest Turing Test experiment to date

#15

Post by pron »

The actual Turing test requires an interrogator interacting with both a human and a machine at the same time, each trying to get the interrogator to declare them the human (and can suggest questions): https://en.wikipedia.org/wiki/Computing_Machinery_and_Intell...

Re: AI21 Labs concludes largest Turing Test experiment to date

#16

Post by TheObviousOne »

tdba wrote: post #3

In summary, humans win the Turing test ~2/3 of the time against current SOTA LLMs. One of the more interesting tactics used was to target a weakness of the LLMs themselves: > ... participants posed questions that required an awareness of the letters within words. For example, they might have asked their chat partner to spell a word backwards, to identify the third letter in a given word, to provide the word that begi…

Interesting tactics might going the other direction.

Asking to generate in a super human capabilities...

write a 65 pages of poem about X...

Re: AI21 Labs concludes largest Turing Test experiment to date

#19

Post by danpalmer »

caddemon wrote: post #12
danpalmer wrote: post #11

Earlier quoted context omitted.

I had a bot send 1 message and then quit. I naturally assumed that bots wouldn't just quit, and 1 message isn't really enough to gauge anything, so yeah based on this experience I wouldn't draw much of a conclusion from it.

Interesting, I had a human quit, surprised bots might quit as well.

I don't know if they discarded any chats that ended before the timer, or without a sufficient number of messages, but that feels important for drawing conclusions.
Post reply on HN