Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Mechanical Turk workers are using AI to automate being human

techcrunch.com

Post reply on HN
Print view View original

31–40 of 169 posts

Re: Mechanical Turk workers are using AI to automate being human

#31

Post by nomilk »

It could be detected like so:

1. Give a sample of mechanical turk workers the same data to classify without telling them

2. Once submitted, look carefully at the ones they get wrong

3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating

4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then they've likely used AI to generate their answers and then fiddled around with a few afterwards to make it look human

That presumes they're fiddling with a few responses for the human touch. If they're submitting raw AI output that should be easily discoverable by running similar models against the same data (like detection of chess cheating compares games against moves form several engines, not just one).

I think that would work, unless everybody uses AI. If that latter were suspected, get a group to label data in person (where you can be 100% sure they can't cheat), then use those answers as the basis for comparison.

Re: Mechanical Turk workers are using AI to automate being human

#32

Post by jvm___ »

nomilk wrote: post #31

It could be detected like so: 1. Give a sample of mechanical turk workers the same data to classify without telling them 2. Once submitted, look carefully at the ones they get wrong 3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating 4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then…

What's the word for maps?

City map makers didn't want people to copy their hard work, so they'd add a short street or some other feature that doesn't actually exist in reality.

Re: Mechanical Turk workers are using AI to automate being human

#33

Post by logicprog »

1323portloo wrote: post #9
bmmayer1 wrote: post #4

This is going to cause problems as many Mechanical Turk tasks are being used to train AI models...

I think this is wonderful. It's a human-bootstrapped AI centipede with predictable, inevitable results.

Twenty fingered monster hands and nonsense garbled word salad here we come! The AI oroborus begins

Re: Mechanical Turk workers are using AI to automate being human

#34

Post by ceejayoz »

jvm___ wrote: post #32
nomilk wrote: post #31

It could be detected like so: 1. Give a sample of mechanical turk workers the same data to classify without telling them 2. Once submitted, look carefully at the ones they get wrong 3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating 4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then…

What's the word for maps? City map makers didn't want people to copy their hard work, so they'd add a short street or some other feature that doesn't actually exist in reality.

https://en.wikipedia.org/wiki/Trap_street

Re: Mechanical Turk workers are using AI to automate being human

#35

Post by nomilk »

jvm___ wrote: post #32
nomilk wrote: post #31

It could be detected like so: 1. Give a sample of mechanical turk workers the same data to classify without telling them 2. Once submitted, look carefully at the ones they get wrong 3. If the turk worker gets similar ones wrong to other turk workers, they're probably not cheating 4. If the turk worker is frequently wrong on questions other turk workers all got right (e.g. if 99.9% of other workers got it right) then…

What's the word for maps? City map makers didn't want people to copy their hard work, so they'd add a short street or some other feature that doesn't actually exist in reality.

> Mapmakers deliberately put slight mistakes in their maps so they can tell when someone copies them. If another map has the same mistake, that's very convincing evidence.

http://www.paulgraham.com/say.html

Re: Mechanical Turk workers are using AI to automate being human

#36

Post by wincy »

TrackerFF wrote: post #5

Are the MT's paid by the hour, or by completed tasks? The latter would obviously incentivize people to use AI. But then again, so would the former, if it means that they can just let some script/program do the work, while they're going on with their day.

It’s paid by task. I used to make $7 an hour doing almost mindless tasks while I was watching TV. But if I went to the bathroom I’d lose out so my effective pay was lower, not to mention I’d get a 1099 on my $300 a year I made or whatever.

Re: Mechanical Turk workers are using AI to automate being human

#37

Post by pixl97 »

mensetmanusman wrote: post #30
sillysaurusx wrote: post #2

This is in fact a precise definition of AGI, by the way. It’s the only true test. If your AI can make money via mturk, you’re on the right path. Superintelligence is when you can make all the money because all the problems are solvable. Right now only some are.

We will know AGI is starting to exist when driverless cars are everywhere.

I'd say that's mid stage AGI. Your latency requirements for driving require the hardware with you. If the early gen AGI is still power hungry then you'll see it plugged into a wall at first.

Re: Mechanical Turk workers are using AI to automate being human

#38

Post by Turskarama »

mensetmanusman wrote: post #30
sillysaurusx wrote: post #2

This is in fact a precise definition of AGI, by the way. It’s the only true test. If your AI can make money via mturk, you’re on the right path. Superintelligence is when you can make all the money because all the problems are solvable. Right now only some are.

We will know AGI is starting to exist when driverless cars are everywhere.

A "general" intelligence is one that can do tasks it wasn't explicitly trained to do. The first self driving cars (that already exist) are powered by an AI that can only drive cars, so it's not general.

Also if self driving cars are everywhere then we won't have AGI, because real AGI largely does away with the need for cars.

Re: Mechanical Turk workers are using AI to automate being human

#39

Post by htrp »

https://news.ycombinator.com/item?id=36432279

Comments from 2 weeks ago.

This research study is using a very simple task, summarization, that is well solved by AI (to the point that everyone points this out as the first use case). Of course, the MTurkers are gonna paste it into ChatGPT. (Also n=44)

Re: Mechanical Turk workers are using AI to automate being human

#40

Post by [deleted] »

TZubiri wrote: post #27
sillysaurusx wrote: post #12

Earlier quoted context omitted.

The invention of nukes caused a similar problem. Scientists had to harvest metal from pre-ww2 sunken battleships, since that’s the only metal not contaminated with some measurable amount of radiation. I forget the details, but it’s very likely pre-chatgpt training data will be prioritized, with exceptions for news and current events (which largely doesn’t matter whether it’s AI generated anyway).

Also happens with wikipedia. Later edits with web references are much more prone to citogenesis, while an early edit with a web reference is probably clear, even if the link is dead. Even printed references can be based on wikipedia now

[deleted]
Post reply on HN