Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% of Google's Emotions Dataset Is Mislabeled

surgehq.ai

Post reply on HN
Print view View original

51–60 of 146 posts

Re: 30% of Google's Emotions Dataset Is Mislabeled

#51

Post by maest »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I wonder how others do this kind of thing.

We did the exact same for a text classification project.

The multi-week grind was awful, but it meant 1. we had a really good understanding of our data 2. we discovered surprising edge cases that we would have missed otherwise.

There is a very large fixed overhead you need to pay when you start outsourcing that work, so doing it yourself is cheaper at scales beyond what you'd normally expect.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#52

Post by carpenecopinum »

alx__ wrote: post #38

Language is hard! Even I, a seasoned native internet dork, have trouble knowing if someone's comment is sarcasm, irony, or something in between. Also, new phrases emerge all the time that turn a phrase on its head, and it has a different emotion. How many feelings can you evoke with a simple, FUCK!

And this is probably going to get even worse the more automatic classification is used to promote or silence content.

A pretty interesting result of this is what I'd call "TikTok speak", where words are replaced, either by similar sounding ones ("porn" => "corn", often times just the corn emoji) or by neologisms ("to kill" => "to unalive"), in the hope of getting around the filters.

This turns natural language on the internet into even more of a moving target than it already used to be.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#54

Post by RabbitUpRabbit »

tgv wrote: post #43
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

This kind of out-sourcing is quite common, and at the prices I heard some time ago, you could run each utterance by 3 to 5 people, which allows you to get an idea of the reliability. But the core of the problem is 27 emotions. That's really asking for trouble.

> But the core of the problem is 27 emotions. That's really asking for trouble.

I'm not sure I agree. The examples highlighted in the blog post aren't cases of slight mislabels (like mislabeling frustration as anger, for example). They are often labeled polar opposite to what they should be.

Though perhaps what you are saying here is that low-paid workers won't bother to look through a list of 27 emotions to find the right one, and thus they are more likely to label at random.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#55

Post by codedokode »

bragr wrote: post #46
stingraycharles wrote: post #41

Earlier quoted context omitted.

The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messag…

>I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying mor…

You can filter out bad workers by preparing an additional well-labeled dataset and removing those who made even a single mistake in it. Also you can give the same task to several workers and check if they give the same label. However this won't protect against a bot using multiple accounts and giving answers based on a hash of a question so that the same question gets the same answer in every account.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#56

Post by Tycho »

codedokode wrote: post #50
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I’m still shocked how low the quality of Mechanical Turk was I don't know about Mechanical Turk, but there is a crowdourcing platform by Yandex. The pay is so low that the reasonable way to earn something is to find a task that is not properly validated and put random answers there from multiple accounts (because there are speed limits). Usually those are tasks by naive foreign companies not knowing about validatio…

Why shouldn’t someone do it for five dollars per hour, and do it properly lest they get fired? Seems like a very easy job and easy to supervise.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#57

Post by alxlaz »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options?

Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent.

The one time I needed anything remotely like this I just took the old "nobody said programming was gonna be glamorous" adage and ran with it for two weeks. Money-wise, two weeks of programmer time sorting data manually sure beats twelve weeks of programmer time debugging models to cope with inaccurate data, and the results are orders of magnitude better.

Given the widespread understanding of how critical training data is, it's mind-boggling to me that businesses in this field try to outsource it to the lowest-paid external company they can find, thus offering the lowest possible performance incentives and pretty much losing any control they have over quality assurance. Then they proceed to spend humongous amounts of money on clean-up and further refining training data sets, with which they could've hired English Lit majors in the first place, who would've given them a perfectly-classified data set from the very beginning.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#58

Post by inertiatic »

HWR_14 wrote: post #31

I've worked on project with more difficult labeling, and we were able to get fairly accurate results. There are tons of standard practices that produce better results, so why did Google ignore them.

Could you point me to resources about best practices in this domain? I've struggled with this and it would potentially help.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#59

Post by alxlaz »

Tycho wrote: post #56
codedokode wrote: post #50

Earlier quoted context omitted.

> I’m still shocked how low the quality of Mechanical Turk was I don't know about Mechanical Turk, but there is a crowdourcing platform by Yandex. The pay is so low that the reasonable way to earn something is to find a task that is not properly validated and put random answers there from multiple accounts (because there are speed limits). Usually those are tasks by naive foreign companies not knowing about validatio…

Why shouldn’t someone do it for five dollars per hour, and do it properly lest they get fired? Seems like a very easy job and easy to supervise.

If it's such an easy job, why outsource it instead of doing it yourself?

Re: 30% of Google's Emotions Dataset Is Mislabeled

#60

Post by olalonde »

hoten wrote: post #27
EarthLaunch wrote: post #9

> LETS FUCKING GOOOOO Could be either anger (let’s fight) or enthusiasm (let’s do it!). Hard problem.

That is 100% enthusiasm, not ambiguous to me at all. But I'm also certain both my parents would read that as antagonistic.

Let's say you are bored out of your mind at a party or while waiting in line for at the bank and tell your friend "LETS FUCKING GOOOOO". It might not be anger but definitely not enthusiasm either.
Post reply on HN