Live data from Hacker News

ChatGPT outperforms crowd-workers for text-annotation tasks

arxiv.org

101–110 of 206 posts

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#101
It does seem to work pretty well. I'm using it to analyze all US Congress bills:

https://govscent.org/bill/USA/118hres190ih

It extracts the topics and determines how on topic the bill is. Soon we're adding a topic browser and the homepage will have some fun stats :) it's all free.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#102
post #79

Earlier quoted context omitted.

Crypto is actually the solution to that. Unlike traditional finance, you don't need a human to sign up under an account. So an AI can just keep it's own wallet and order humans to set up server farms.

If AI ends up being the one finding a real usecase for crypto, will we have the indisputable proof that it is smarter than humans? :)

Or maybe, AI will notice how profitable extortion is and approach it with algorithmic efficiency.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#103
post #65

Earlier quoted context omitted.

The market disagrees with you. How come there are billions of dollars spent on all these knowledge workers around the world every day when they could be replaced by this expert-level AI? I'm not sure where this idea of LLMs being intelligent even comes from. It took me a whopping 9 prompts (genuine questions, no clever prompt engineering) of interacting with ChatGPT to conclude it does not understand anything . It do…

To give an example of the limitations of these things that's hopefully easy to understand, I got access to Bard this morning and asked it to write a limerick. It gave me what could charitably be called a free verse poem that happened to begin "there once was a man from Nantucket." I'm sure they can improve on it (ChatGPT was better at this kind of thing when I had access to it) but "solved problem" is clearly a long…

Seems Pretty Good to me! Better than I could do anyway. Bard is a joke compared to GPT-4: "Write a limerick about a dog"

  There once was a dog from the pound
  Whose bark had a curious sound
  With a wag and a woof,
  He'd jump on the roof,
  Delighting the folks all around.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#104

Earlier quoted context omitted.

Eh, even if the probability distribution is unknown, a random guess should still have 33% chance to be correct.

I started explaining why you were wrong and got about 5 words in before I realized you're correct.

Probability: always counterintuitive

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#105
post #91
post #82

Earlier quoted context omitted.

Not sure why you got downvotes. It seems like it's a solved issue indeed. AI reasoning has still some way to go, but it seems language understanding is a finished subject.

Regurgitating training data trigram by trigram is not how human language processing works.

You sure about that? The more I interact with LLMs and learn how they operate, the more it seems to me like people operate on very similar principles and algorithms with their use of language.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#106

Earlier quoted context omitted.

To give an example of the limitations of these things that's hopefully easy to understand, I got access to Bard this morning and asked it to write a limerick. It gave me what could charitably be called a free verse poem that happened to begin "there once was a man from Nantucket." I'm sure they can improve on it (ChatGPT was better at this kind of thing when I had access to it) but "solved problem" is clearly a long…

Seems Pretty Good to me! Better than I could do anyway. Bard is a joke compared to GPT-4: "Write a limerick about a dog" There once was a dog from the pound Whose bark had a curious sound With a wag and a woof, He'd jump on the roof, Delighting the folks all around.

Yes, much more compelling. But if this were a “solved problem” then any of them should be able to do it easily. It’s not like I need to compare the results of sorting between different programs. It just works. That is a solved problem.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#108
post #20

My main take away here is that Turkers are terrible at some of these tasks. The "stance" task is, "Classify the tweet as having a positive stance towards Section 230, a negative stance, or a neutral stance.", and the Turkers accuracy was like 20%. Even in its best task, ChatGPT only got 75% accuracy.

I have long suspected that Turkers dishonestly perform the tasks. At 0.06 cents per task, you're really incentivizing "finish the task as quickly as possible"; and "press the left button" is a lot faster than "read the tweet, think about it, and classify".

You always use multiple MTurks on same job when you use them.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#109
post #12

Earlier quoted context omitted.

Also "the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk." It's interesting that the best available MTurk Master crowd-workers located in the US are paid about six cents per task.

I guess my surprise is that the machine is only 20x cheaper than the cheapest human available.

ChatGPT costs the same, regardless of the difficulty of the task, which is not the case for humans.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#110

Earlier quoted context omitted.

I have long suspected that Turkers dishonestly perform the tasks. At 0.06 cents per task, you're really incentivizing "finish the task as quickly as possible"; and "press the left button" is a lot faster than "read the tweet, think about it, and classify".

You always use multiple MTurks on same job when you use them.

OK, but if all MTurks are doing this (because why wouldn't they) then you are just sampling a random variable.
Post reply on HN