Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT outperforms crowd-workers for text-annotation tasks

arxiv.org

Post reply on HN
Print view View original

171–180 of 206 posts

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#171

Post by intended »

jacknews wrote: post #16

So is this 'AI trains AI better than people can'? And presumably the better-trained AI will also be better again at training. I think I've seen this movie.

No its not.

This is

High quality training data set made by trained humans. (Ground truth)

Chat GPT was given the rules and data set. then asked to evaluate the data.

MTurkers were given the rules and data set. Then asked to evaluate the data.

GPT was closer to the gold standard/Ground truth.

If you want a historical comparison, this is industrialization. Before we could only scale using humans via mturk.

Now if you can scale using GPT.

However, for ALL of GPT related shenanigans -

1) This is English First. You cannot apply this to under-resourced languages.

2) You need to have the ability to verify the output.

——-

I really want to solve 1.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#172

Post by riku_iki »

xyzzyz wrote: post #35
quickthrower2 wrote: post #32

Earlier quoted context omitted.

There are 3 stances. But 33% is achievable by chance.

Only if probability distribution of answers is known.

just speculating, there is a chance ChatGPT was trained on this dataset in the past, and learned probability distribution..

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#173

Post by Minor49er »

malshe wrote: post #167
YeGoblynQueenne wrote: post #122

Suppose you have two classifiers, A and B, and some un-annotated data, D. You want to know how good is classifier B at annotating the data, compared to classifier A. One problem is that you don't have the ground truth for D. So you start by annotating D with the labels assinged by a third classifier, C: C(D) → D₁ Having thus established a modicum of "ground truth", ish, you proceed to annotate D with the two classifi…

Your comment is based on a very strong assumption that all human annotators are alike in their motivations and abilities to do the tasks. As someone who has used mTurk in the past quite a lot, I think this assumption is wrong. That was the reason I stopped using mTurk. On a separate note, how do you type those arrows and subscripts in an HN comment?

This is true. I set up a data validation project with mTurk in the past to validate scraped data by presenting a simply survey of the results. Basically "go to this webpage. The title is X (True or False). The description is Y (True or False)" etc. There were several users who would speedrun the surveys which created a lot of false positives

mTurk has some ways to make results more accurate, such as being able to programmatically tank users who have given bad results and making it easy to give bonuses to users who give good results. But those have to be carefully designed within the application

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#174

Post by danpalmer »

kiratp wrote: post #169
danpalmer wrote: post #158

At my previous job we had a human review stage in a data pipeline. 5-10 people at an outsourcing company in Bangladesh would review things via a simple web interface we provided for them. There were ~10 factors they were reviewing, all fixed options (no free text), but varying from 5 to 500 options per factor. It was all based on a few text fields and around 5 images. On the surface of it, I'd expect ChatGPT to do ve…

LangChain + a vector DB for embedding [0] sections of your doc would solve the problem today. You could also have failsafes that trigger human oversight based on confidence levels or other factors. [0] https://python.langchain.com/en/latest/modules/indexes/getti...

It looks like for it to work like this the doc would need to be divisible into obvious units that could be retrieved by some key/terms, but in our case it was much looser than that. There were no fixed rules (otherwise we'd have written business logic), it was guidance that really assumed the whole doc as context, and human judgement of the relative weightings of parts of the doc.

The failsafes based on confidence levels is an interesting idea though, that is very possible. I suspect we'd have started out with manually reviewing all decisions and slowly backed it off as the prompts got better.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#175

Post by halfeatenscone »

karmasimida wrote: post #144

It should comes with no surprise to anyone. Text classification is an easy task. ChatGPT is definitely overqualified to perform this task.

"Classify the following strings according to whether they represent Python programs which terminate..."

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#176

Post by tgv »

maxdoop wrote: post #133
tgv wrote: post #91

Earlier quoted context omitted.

Regurgitating training data trigram by trigram is not how human language processing works.

And how does it work, then? Everyone shrugs and says, “nope, humans are different”. I’ve commented about 100 times recently asking for detail as to how human language / thought works, yet have not seen an answer.

We interpret what we hear, make a mental representation of that (incrementally; this process sometimes fails), which links to concepts, which in turn can link to memories, then "look for the answer" (if it's a question) by association and puzzling, the former is pretty quick, the latter slow, check if the answer makes sense, and formulate a reply. We can start formulating a reply from similarly formed structures while completing the thought, because we monitor our speech. When that happens, you often say "er..."

That's basic linguistics and cognitive psychology. Nothing an LLM has done has invalidated that.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#177

Post by tgv »

hallway_monitor wrote: post #105
tgv wrote: post #91

Earlier quoted context omitted.

Regurgitating training data trigram by trigram is not how human language processing works.

You sure about that? The more I interact with LLMs and learn how they operate, the more it seems to me like people operate on very similar principles and algorithms with their use of language.

The "if it walks like a duck" school of ontology.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#178

Post by elif »

danpalmer wrote: post #158

At my previous job we had a human review stage in a data pipeline. 5-10 people at an outsourcing company in Bangladesh would review things via a simple web interface we provided for them. There were ~10 factors they were reviewing, all fixed options (no free text), but varying from 5 to 500 options per factor. It was all based on a few text fields and around 5 images. On the surface of it, I'd expect ChatGPT to do ve…

But for 20x cost savings, you could have 18 different adversarial models criticizing and correcting the output of other models and still come out ahead financially. I imagine with multiple "sets of sets of eyes" the edge cases would bubble up and could have a model specific to handle those edge cases.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#179

Post by ftxbro »

CGamesPlay wrote: post #98
Imnimo wrote: post #20

My main take away here is that Turkers are terrible at some of these tasks. The "stance" task is, "Classify the tweet as having a positive stance towards Section 230, a negative stance, or a neutral stance.", and the Turkers accuracy was like 20%. Even in its best task, ChatGPT only got 75% accuracy.

I have long suspected that Turkers dishonestly perform the tasks. At 0.06 cents per task, you're really incentivizing "finish the task as quickly as possible"; and "press the left button" is a lot faster than "read the tweet, think about it, and classify".

> At 0.06 cents per task

Do you work at Verizon?

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#180

Post by elif »

ftxbro wrote: post #113
andruby wrote: post #112

Earlier quoted context omitted.

When will we see a form of "arbitrage" where someone uses ChatGPT to do the work of an MTurk and pockets the difference? Will that lead to MTurk prices converging to ChatGPT prices?

Even before LLMs, MTurk has been in a war with the botters, and my understanding is that the MTurkers have to periodically do captchas but many still use bots or various automated tools or utilities. The LLMs and especially the multimodal ones will only make the bots even better and break the captchas even harder. MTurk's business model specifically wants human workers categorized into fine grained categories so they…

Is there such a thing as a captcha in the age of GPT4?

Maybe openAI put safety barriers on their model... But surely the botters have a model without that by now?

Post reply on HN