Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT outperforms crowd-workers for text-annotation tasks

arxiv.org

Post reply on HN
Print view View original

201–206 of 206 posts

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#201

Post by YeGoblynQueenne »

sebzim4500 wrote: post #139
YeGoblynQueenne wrote: post #124

Earlier quoted context omitted.

Since I've been seeing this wild fantasy being bandied about for a while now, I have to point out that, if you had "AI" that could train "AI", you wouldn't need to train any more "AI". Because at that point, there would be nothing to gain. Suppose you have a text classifier that can produce text classifications just as good as those of human annotators, so that you could use it to train other text classifiers. At tha…

>if you had "AI" that could train "AI", you wouldn't need to train any more "AI". Because at that point, there would be nothing to gain. Not necessarily, the AI used to classify might be much more expensive to run than the new AI you are training. This is what Tesla does, for example. They have a massive model which they use as a 'ground truth' to train the models which can actually run in the car.

Well, OK, there are gains in efficiency, but the context here is, as far as I understand it, letting an "AI" train a better "AI".

Perhaps I misunderstand the OP and like I say in my comment above, I've seen comments to that extent pop up here and there om HN in the last few days, so maybe I'm jumping to conclusions about what the OP meant. I should have asked for clarifications (although usually when I do there's no response; not from the OP, from most users).

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#202

Post by YeGoblynQueenne »

nico wrote: post #165
YeGoblynQueenne wrote: post #124

Earlier quoted context omitted.

Since I've been seeing this wild fantasy being bandied about for a while now, I have to point out that, if you had "AI" that could train "AI", you wouldn't need to train any more "AI". Because at that point, there would be nothing to gain. Suppose you have a text classifier that can produce text classifications just as good as those of human annotators, so that you could use it to train other text classifiers. At tha…

> Note that both language models and image classifiers have been "beating" human performance in benchmarks for a while now, and still they are not used to train other classifiers. Mmhh have you seen Alpaca? It’s the Llama model (Meta’s), trained (fine-tuned) using GPT. Yesterday there was a paper showing ChatGPT is better than mechanical turks. > the ground truth is always the decisions made by humans But why does AI…

I replied the way I replied because I assumed you meant "AI" training better "AI". Please help me understand if that's what you really meant, because if that's what you meant then, yes, that is a complete fantasy.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#203

Post by YeGoblynQueenne »

nmca wrote: post #200
YeGoblynQueenne wrote: post #189

Earlier quoted context omitted.

To be clear, that's a different criticism of the article's methodology than mine, yes? If you assume that the two sets of human annotators are different, then what are you comparing, exactly? The ability of one group to second-guess the other? There are other issues if you choose to assume that the two groups are fundamentally dissimilar: one group was two grad students, the other a number of Mechanical Turks. You ca…

In the framing of your original comment: A: Mturkers B: ChatGPT C: Experts => ChatGPT better approximates the labelling of D by experts than mTurkers Which is a coherent and interesting conclusion. Edit: also, please forgive the snark in my first response

>> Edit: also, please forgive the snark in my first response

No need to apologise! Your snark wasn't overboard, I thought. Anyway, big girl, can take it :)

>> ChatGPT better approximates the labelling of D by experts than mTurkers

That could be a "coherent and interesting conclusion" but it's not what the article really claims. The article's title is I think hedging its bets, by being very precise about who, exactly, was outperformed by ChatGPT, although it still manages to be vague about how ChatGPT outperformed the Mechanical Turks.

I'm also really doubtful that two political science graduate students can be considered as "experts" in the annotation tasks they were called to perform, which were, again if I got that right, about content moderation. "Experts" in this setting would be people with experience in moderating discussion boards etc. I don't see that this was the case with the two students that provided the initial annotation.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#204

Post by lisasays »

jillesvangurp wrote: post #61
lisasays wrote: post #39

Earlier quoted context omitted.

I can't vouch for the correctness obviously. Therein lies the rub. There's a huge gap between what LLMs can currently do (spit back something in a target language that gives you the basic idea, however awkwardly phrased, of what was said in the source language). And what is actually needed for idiomatic, reasonably error-free translation. By "reasonably error-free" I mean, say, requiring a human correction for less t…

I've tried it between English and Dutch (which is my native language). It's pretty fluent, makes less grammar mistakes than google translate and seems to generally get the gist of the meaning across. It's not a pure syntactical translation. Which is why it can work even between some really obscure language pairs. Or indeed programming languages. Where it goes wrong is when it misunderstands context. It's not an AGI a…

Thanks for counter-example; I'll confess to having spent far too much time with edge-case translations of late (on languages a bit farther apart), rather than on more generic cases like the above.

I will be re-assessing my view on general-case translation performance accordingly.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#205

Post by sorz »

sorz wrote: post #3

Is NLP a solved problem now?

Just watched a talk[0] about natural language understanding research in post-GPT-3 era. Old issues may has been solved, while new topics are coming to this area (quoted from the slides):

- Retrieval augmented in-context learning

- Better benchmarks

- Last mile for productive application

- Faithful, human-interoperable explanations

[0] https://www.youtube.com/watch?v=-lnHHWRCDGk

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#206

Post by winrid »

Ajedi32 wrote: post #152
winrid wrote: post #101

It does seem to work pretty well. I'm using it to analyze all US Congress bills: https://govscent.org/bill/USA/118hres190ih It extracts the topics and determines how on topic the bill is. Soon we're adding a topic browser and the homepage will have some fun stats :) it's all free.

Wow, I didn't realize how many congressional bills are just pointless resolutions with zero legislative impact. Is this list of bills curated in any way? Where are you sourcing it from? A quick scan of https://www.govinfo.gov/app/collection/bills/ seems to turn up bills with a lot more substance. Edit: Upon further investigation it seems like a lot of those are not technically bills, but rather House or Senate resolu…

Update - long bills are now supported: https://govscent.org/bill/USA/111s570pcs
Post reply on HN