Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT outperforms crowd-workers for text-annotation tasks

arxiv.org

Post reply on HN
Print view View original

51–60 of 206 posts

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#51

Post by nico »

famouswaffles wrote: post #15
rjh29 wrote: post #9

Earlier quoted context omitted.

Idiomatic translation of text matching a human professional (e.g. free of errors for legal terms, interesting and natural for fiction) is unlikely to be achieved until we have AGI. So no.

I won't comment on the first bit as i've not personally tested in that are but GPT-4 can absolutely make short work on the second. I don't think people realize how good Bilingual LLMs are at translations. Yes you have idioms transfer between languages. Feel free to test it yourself.

> I don't think people realize how good Bilingual LLMs are at translations

This.

GPT/ChatGPT is able to even translate between different "accents" or dialects of the same language. For example it can give you the same sentence in Mexican, Argentinean or Peruvian Spanish.

Example:

Me: Give me a sentence in spanish that is different in Mexican, Argentinean and Peruvian Spanish. Write the sentence in each dialect.

ChatGPT: Mexican Spanish: "¿Qué onda, güey? ¿Cómo estás?" Argentinean Spanish: "¿Qué onda, boludo? ¿Cómo estás?" Peruvian Spanish: "¿Qué tal, causa? ¿Cómo estás?"

These sentences mean "What's up, dude? How are you?" in English. The primary difference is the slang term used for "dude" or "friend" in each dialect: "güey" in Mexican Spanish, "boludo" in Argentinean Spanish, and "causa" in Peruvian Spanish.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#53

Post by pornel »

rtkwe wrote: post #27
sorz wrote: post #3

Is NLP a solved problem now?

I still think we're a long ways off. LLMs can't to my knowledge process a request into a lookup on say an actual database of facts at the moment or parse a request into API actions. So far it's shown it's really really good at continuing a conversation with more text but as far as I understand them there's not a usable comprehension of what's actually being asked and answered. The point that would say to me the LLM a…

> LLMs can't to my knowledge process a request into a lookup on say an actual database of facts at the moment or parse a request into API actions.

They can: https://arxiv.org/abs/2302.04761

> In this paper, we show that LMs can teach themselves to use external tools

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#54

Post by famouswaffles »

shock-value wrote: post #44
famouswaffles wrote: post #8

Earlier quoted context omitted.

We have artificial intelligence that is general and above average human intelligence for the majority of tasks it can perform. Near expert level for some. NLP is a solved problem. Bespoke models are out the door. Large enough LLMs crush anything else for any NLP task. Honestly, this whole "they are not intelligent" argument is becoming ridiculous. might as well argue that a plane isn’t a real bird or a car isn’t a re…

The debate over what kind of intelligence these models possess is rightly lively and ongoing. It’s clear that at the least, they can decipher very numerous patterns across a wide range of conceptual depths — it’s an architectural advance easily on the the level of the convolutional neural network, if not even more profound. The idea that NLP is “solved” isn’t a crazy notion, though I won’t take a side on that. That s…

AGI is Artificial General Intelligence. We have absolutely passed the bar of artificial and generally intelligent. It's not my fault goal post shifting is rampant in this field.

And you want to know the crazier thing? Evidently a lot of researchers feel similarly too.

General Purpose Technologies ( from the Jobs Paper), General Artificial Intelligence (from the creativity paper). Want to know the original title of the recent Microsoft paper ? "First contact with an AGI system".

The skirting around the word that is now happening is insanely funny. Look at the last one. Fuck, they just switched the word order. Nobody wants to call a spade a spade yet but it's obvious people are figuring it out.

I can you show you output that clearly demonstrates understanding and reasoning. That's not the problem. The problem is that when I do, the argument Quickly shifts to "it's not true understanding!" What a bizzare argument.

This is the fallacy of the philosophical zombie. Somehow there is this extra special distinction between two things and yet you can't actually show it. You can't test for so called huge distinction. A distinction that can't be tested for is not a distinction.

The intelligence arguments are also stupid because they miss the point entirely.

What matters is that the plane still flies, the car still drives and the boat still sails. For the people who are now salivating at their potential, or dreading the possibility of being made redundant by them, these large language models are already intelligent enough to matter.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#56

Post by zquzra »

nico wrote: post #51
famouswaffles wrote: post #15

Earlier quoted context omitted.

I won't comment on the first bit as i've not personally tested in that are but GPT-4 can absolutely make short work on the second. I don't think people realize how good Bilingual LLMs are at translations. Yes you have idioms transfer between languages. Feel free to test it yourself.

> I don't think people realize how good Bilingual LLMs are at translations This. GPT/ChatGPT is able to even translate between different "accents" or dialects of the same language. For example it can give you the same sentence in Mexican, Argentinean or Peruvian Spanish. Example: Me: Give me a sentence in spanish that is different in Mexican, Argentinean and Peruvian Spanish. Write the sentence in each dialect. ChatG…

I don't recommend you call random people "boludo" in Argentina.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#57

Post by gwoolhurme »

famouswaffles wrote: post #15
rjh29 wrote: post #9

Earlier quoted context omitted.

Idiomatic translation of text matching a human professional (e.g. free of errors for legal terms, interesting and natural for fiction) is unlikely to be achieved until we have AGI. So no.

I won't comment on the first bit as i've not personally tested in that are but GPT-4 can absolutely make short work on the second. I don't think people realize how good Bilingual LLMs are at translations. Yes you have idioms transfer between languages. Feel free to test it yourself.

I am also multilingual as well and I've tested it personally. English Portuguese does really well, but Portuguese Japanese or even Japanese English is not as good as a human translator by a long shot because of a lot of hidden subtext in conversation. Even something that a university student would probably pickup on in their first year of Japanese as a foreign language. It is still much better than GPT-3.5, so much so that it made a lot of waves here in Japan, but a few friends who work in translation of books and manga find it is not really a go-to tool yet (yet...).

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#58

Post by nico »

zquzra wrote: post #56
nico wrote: post #51

Earlier quoted context omitted.

> I don't think people realize how good Bilingual LLMs are at translations This. GPT/ChatGPT is able to even translate between different "accents" or dialects of the same language. For example it can give you the same sentence in Mexican, Argentinean or Peruvian Spanish. Example: Me: Give me a sentence in spanish that is different in Mexican, Argentinean and Peruvian Spanish. Write the sentence in each dialect. ChatG…

I don't recommend you call random people "boludo" in Argentina.

It really depends on the tone and context. If you are a tourist and say it in a joking manner, people are probably going to laugh. If you say it in anger to someone, they might not like it very much.

Similar to how a lot of swear words work in many languages.

It's interesting to see how what matters is not the word, but the intention behind it. At the end we are trying to communicate meaning, and words are just one of our tools to do it.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#59

Post by DJBunnies »

nico wrote: post #38
jacknews wrote: post #16

So is this 'AI trains AI better than people can'? And presumably the better-trained AI will also be better again at training. I think I've seen this movie.

That's exactly what's coming. AI will train AI. At some point AI is going to stop needing us to keep evolving. Not sure what that will be like.

Call me when server farms can reproduce on their own, and network together, and source electricity.

Re: ChatGPT outperforms crowd-workers for text-annotation tasks

#60

Post by famouswaffles »

gwoolhurme wrote: post #57
famouswaffles wrote: post #15

Earlier quoted context omitted.

I won't comment on the first bit as i've not personally tested in that are but GPT-4 can absolutely make short work on the second. I don't think people realize how good Bilingual LLMs are at translations. Yes you have idioms transfer between languages. Feel free to test it yourself.

I am also multilingual as well and I've tested it personally. English Portuguese does really well, but Portuguese Japanese or even Japanese English is not as good as a human translator by a long shot because of a lot of hidden subtext in conversation. Even something that a university student would probably pickup on in their first year of Japanese as a foreign language. It is still much better than GPT-3.5, so much s…

Oh for sure i don't mean to say it's excellent in every language. But i personally think a lot of that is training data representation. Doesn't need to be anywhere equal but for instance after English(93%), the biggest language representation for GPT-3's training corpus is french at...1.8 % Pretty wild.

Of Course i don't know the data for GPT-4

Post reply on HN