ChatGPT outperforms crowd-workers for text-annotation tasks
1–10 of 206 posts
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#2> We used the ChatGPT API with the ‘gpt-3.5-turbo’ version to classify the tweets.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#3Is NLP a solved problem now?
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#4Is NLP a solved problem now?
for stuff that before 2021-09, mostly.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#5Is NLP a solved problem now?
Obviously not. If we had a general solution to language intelligence we would have artificial intelligence at the level of at least human intelligence – which we do not. Rather, the right question to ask is which language intelligence tasks currently have acceptable performance and under which conditions (text domain, etc.). Clearly this is a much more difficult question and with a lot more nuance to it, even if it is undeniably that things have moved very quickly over the last few years.
Skimming the abstract as a senior academic in the area. This looks like preliminary work and a limited investigation for a single (non-standard) task. Thus far from a strong result published at say a top-tier conference or journal. Still, interesting direction and if expanded upon could absolutely be impactful. I should also mention that I am not familiar with the related literature, so it could very much be that there is similar (better?) work out there exploring the same question.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#6Is NLP a solved problem now?
Mostly, yes. LLMs will turn natural language into really whatever form you want.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#7Before you ask, because I was curious, from the paper:
"For MTurk, we aimed to select the best available crowd-workers, notably by filtering for workers who are classified as “MTurk Masters” by Amazon, who have an approval rate of over 90%, and who are located in the US."
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#8Is NLP a solved problem now?
Obviously not. If we had a general solution to language intelligence we would have artificial intelligence at the level of at least human intelligence – which we do not . Rather, the right question to ask is which language intelligence tasks currently have acceptable performance and under which conditions (text domain, etc.). Clearly this is a much more difficult question and with a lot more nuance to it, even if it…
We have artificial intelligence that is general and above average human intelligence for the majority of tasks it can perform. Near expert level for some. NLP is a solved problem. Bespoke models are out the door. Large enough LLMs crush anything else for any NLP task.
Honestly, this whole "they are not intelligent" argument is becoming ridiculous.
might as well argue that a plane isn’t a real bird or a car isn’t a real horse.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#9Is NLP a solved problem now?
Idiomatic translation of text matching a human professional (e.g. free of errors for legal terms, interesting and natural for fiction) is unlikely to be achieved until we have AGI. So no.
Re: ChatGPT outperforms crowd-workers for text-annotation tasks
#10Before you ask, because I was curious, from the paper: "For MTurk, we aimed to select the best available crowd-workers, notably by filtering for workers who are classified as “MTurk Masters” by Amazon, who have an approval rate of over 90%, and who are located in the US."
honestly i wonder if @dang will approve an auto summarizer bot on HN since it helps improves the quality of discussions. finetune on HN comments, anticipate the top few questions, and then answer from the source doc