Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

artisana.ai

Post reply on HN
Print view View original

51–60 of 151 posts

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#51

Post by margorczynski »

brian_spiering wrote: post #6
famouswaffles wrote: post #4

NLP is solved, more or less. Either way, Bespoke NLP is on its way out. It's pretty funny how buried this is in the original paper.

Parts of NLP have made great progress. There are still parts of NLP that could still be improved, such as the truthiness of generated answers.

But that's not an NLP problem at heart. Language is just a collections of tokens (words, letter) that are tied together by certain rules to convey some meaning. There is no concept of reality per se.

For example, consider filling the blank:

A giant ______ flew over my head!

It can be a plane. Or a dragon. Or an UFO. Or a balloon. The thing is all of those are correct answers language-wise and the model works correctly as long as what gets filled in conforms to the rules of the given language.

The language that we generate encodes reality to some extent and the model picks up those correlations but there is no concept of reasoning or reality behind it. Maybe it is emergent at some point (as to effectively compress it needs to encode some subset of rules governing our reality) but it is not an agent that optimizes for understanding our reality. Something like Dreamer would be much closer to that.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#52

Post by fshbbdssbbgdd »

troops_h8r wrote: post #7

I don't think I see enough discussion about what this means for privacy. There was some protection in the fact that it was prohibitively expensive to get someone to listen to every single one of our phonecalls/read all our emails/etc. Worrying that this will no longer be the case.

With regard to privacy, what’s the difference between your email’s text stored on a server, and your email’s text alongside the output of the text processed through a LLM? If “they” can already look at the text, what more privacy is there to lose?

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#53

Post by nr2x »

876978095789789 wrote: post #50

Great to see this tech and the money invested in it being used to take low-paying jobs away from people with limited options, instead of something like drug discovery or cancer biology.

In this case it is academic research released to the public, which is why anybody knows about it. This is a fairly good thing compared to any alternative I can think of.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#54

Post by 876978095789789 »

nr2x wrote: post #53
876978095789789 wrote: post #50

Great to see this tech and the money invested in it being used to take low-paying jobs away from people with limited options, instead of something like drug discovery or cancer biology.

In this case it is academic research released to the public, which is why anybody knows about it. This is a fairly good thing compared to any alternative I can think of.

> In this case it is academic research released to the public

That's going to be the exception, not the rule. The benefits from automating crowdwork will disproportionately accrue to corporate profits.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#55

Post by selectodude »

boringuser2 wrote: post #31
local_crmdgeon wrote: post #21

Earlier quoted context omitted.

This is a problem that money can solve.

If money could solve this problem, China would lead AI, not OpenAI.

200,000 wafers per month is a lot of GPUs.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#56

Post by nr2x »

stevenhuang wrote: post #42
alphanumeric0 wrote: post #18

Earlier quoted context omitted.

Why do you think there was a mass surveillance of American domestic communications since forever ago, as leaked by Snowden? This technology has been available since then and can effectively summarize millions of pieces of communication.

> This technology has been available since then No, it hasn't lol.

How sure are you about that? The basic theories have been around since the 70s, have been proven at scale in the last decade, and the NSA has more data and compute than anybody else. I’d be shocked if they aren’t very far along in solving many problems.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#57

Post by ftxbro »

If you look at the table, the GPT-4 model has better correlation with the expert ensemble than the crowd does, but only on some criteria. The GPT-4 model is closer for all of the ethics questions, but the crowd is closer for the utility level and economic impact questions.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#58

Post by hnaouesteuho »

From reading the paper, GPT-4 also outperformed the researchers themselves in many categories, despite the researchers being the ones who created the dataset being used to perform the comparison.

In other words, the metrics are biased in the researchers’ favor — so GPT-4 would have beat them even more often (probably a majority of the time based on the numbers), if someone else had created the guidelines and golden labels.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#59

Post by lightedman »

stevenhuang wrote: post #42
alphanumeric0 wrote: post #18

Earlier quoted context omitted.

Why do you think there was a mass surveillance of American domestic communications since forever ago, as leaked by Snowden? This technology has been available since then and can effectively summarize millions of pieces of communication.

> This technology has been available since then No, it hasn't lol.

Yes, it has. Consumer-wise, we've had Dragon Naturally Speaking since the late 90s. It's pretty simple to have a script read what it outputs text-wise and look for key words. No AI is even needed to do this.

Re: GPT-4 Outperforms Elite Crowdworkers, Saving Researchers $500k and 20k hours

#60

Post by nr2x »

mztwo wrote: post #46
malux85 wrote: post #23

Earlier quoted context omitted.

Yeah but have you seen the leaked slides? It's clear that they have only the ability to analyse 10% or less of the data they are storing . GPT-like systems will close that gap, and then comes all of the problems of automated law enforcement - Extrapolation from incomplete data, false positives from coindicences, interpretation errors, all that annoying stuff

To add to that, the leaked documents from Snowden described a query language not unlike doing Boolean searches. Nothing close to GPT’s ability to comprehend human asks.

Yes but you can get GPT to write structured queries based on natural language. It works very, very well at turning normal phrases into SQL.
Post reply on HN