Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% of Google's Emotions Dataset Is Mislabeled

surgehq.ai

Post reply on HN
Print view View original

41–50 of 146 posts

Re: 30% of Google's Emotions Dataset Is Mislabeled

#41

Post by stingraycharles »

shreyshnaccount wrote: post #19
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clustering based op and re-label then clusters? does that even make sense?

The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random.

I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible.

What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messages based on certain dimensions: eg “contradictions” (“positive” and “negative” being closest as possible, or “angry” and “happy”), “least” (it was surprisingly difficult to find positive and uplifting tweets, and you don’t want a dataset with 90% negative messages!), etc.

That way we worked our way through the dataset and were able to get a pretty decent dataset in about a week time.

No idea how others approach this type of problem, but it’s what I came up with.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#42

Post by stingraycharles »

echen wrote: post #12
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a startup solving this exact problem ( https://www.surgehq.ai ), and previously built the human computation platforms at a couple FAANGs (precisely because this was a huge issue I always faced internally). We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis,…

Good news for you: being your target audience, we actually did have you guys on our radar.I

For the scale of our project, however, the price point was prohibitive.

We ended up building a small cli tool that interactively trained the model, and allowed us to focus on the most important messages (eg those where positive/negative sentiment was closest, the labels with the smallest volume, etc).

EDIT: If I now look at your website, it seems like you’ve also just provide good tooling for doing these types of things yourself? If that were the case, I wouldn’t mind having paid $50-$100 for a week of access to such a tool. But $20/hr to hire someone who classifies data which we would still need to audit afterwards was too much for us.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#43

Post by tgv »

nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

This kind of out-sourcing is quite common, and at the prices I heard some time ago, you could run each utterance by 3 to 5 people, which allows you to get an idea of the reliability.

But the core of the problem is 27 emotions. That's really asking for trouble.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#44

Post by david_draco »

pmontra wrote: post #32
Thorrez wrote: post #17

Earlier quoted context omitted.

>Farm it out to low-paid non-native speakers The paper claims: >“All raters are native English speakers from India.”

US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.

Is the "right answer" a classification based on US English with background knowledge of US cultural background, or is the goal to build a global sentiment data set?

You and OP ("the indians labelers don't know how to do it correctly") seem to want the former, so it would be good to state that goal upfront.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#45

Post by COMMENT___ »

CurleighBraces wrote: post #40
COMMENT___ wrote: post #23

Earlier quoted context omitted.

I don't see anger in "LETS F**ING GOOOOO". It's just a comment that says "let's do X" in impatient and enthusiastic manner.

How about, "LETS F*ING GOOOOO YOU DINGBAT", now if that's a comment between friends the addition of the insult might be said in jest and still be impatient/enthusiastic, or does by adding the insult to the end automatically label it as combative? I realise this wasn't part of the dataset, more making a point that written language without context ( and sometimes even with ) is subject to huge amounts of reader interpr…

This one is indeed hard to label.

PS Hope they will never begin solving philosophy problems with ML.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#46

Post by bragr »

stingraycharles wrote: post #41
shreyshnaccount wrote: post #19

Earlier quoted context omitted.

you have experience with this, so you're probably the perfect person to ask this to- why didn't you just use the old (inaccurate) labels, perform some clustering based op and re-label then clusters? does that even make sense?

The old (inaccurate) labels were so completely utter shit (pardon my French) that it may as wel have been random. I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. What I ended up doing was make some Python CLI-based took that made it extremely fast for us to classify messages; after seeding it with about 1000 classifications, we would then focus on the messag…

>I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible.

This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying more could overcome that because so many people engage with the platform in bad faith.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#47

Post by jackbrookes »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options?

The way I think Google does it with reCAPTCHA is to request classification for each item multiple times. If they differ, keep sending them out until you get a consensus on those items. It weeds out those responses that were just basically random clicks.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#48

Post by notahacker »

xorcist wrote: post #30
hoten wrote: post #27

Earlier quoted context omitted.

That is 100% enthusiasm, not ambiguous to me at all. But I'm also certain both my parents would read that as antagonistic.

It could also be 100% irony. Without context it's hard to tell.

Which applies to a lot of online content, especially on Reddit

Kamala 2020!!!! could be a ringing endorsement, a lighthearted parody of ringing endorsement or utter derision depending on context so I'm not even sure the commenter classifying it as "neutral" was wrong.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#49

Post by echen »

CurleighBraces wrote: post #40
COMMENT___ wrote: post #23

Earlier quoted context omitted.

I don't see anger in "LETS F**ING GOOOOO". It's just a comment that says "let's do X" in impatient and enthusiastic manner.

How about, "LETS F*ING GOOOOO YOU DINGBAT", now if that's a comment between friends the addition of the insult might be said in jest and still be impatient/enthusiastic, or does by adding the insult to the end automatically label it as combative? I realise this wasn't part of the dataset, more making a point that written language without context ( and sometimes even with ) is subject to huge amounts of reader interpr…

Agree that context is often needed! (Which is why it was strange to us that raters weren't presented with any context besides the comment text itself -- not even the subreddit, much less the original Reddit post.)

One interesting question, though: if "LETS FUCKING GOOOOO YOU DINGBAT" were meant to be a combative insult, would someone still add a bunch of O's ("GOOOOO" instead of merely "go")? My intuition is that if combativeness were intended, "let's fucking go, you dingbat" would be more likely than "LETS FUCKING GOOOOO YOU DINGBAT", but of course it's a bit hard to say without that context.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#50

Post by codedokode »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I’m still shocked how low the quality of Mechanical Turk was

I don't know about Mechanical Turk, but there is a crowdourcing platform by Yandex. The pay is so low that the reasonable way to earn something is to find a task that is not properly validated and put random answers there from multiple accounts (because there are speed limits). Usually those are tasks by naive foreign companies not knowing about validation.

So if you want high quality you need to implement proper validation, triple check every label by different people and do not expect that someone is going to do it for $5/hour. And maybe you should learn how a crowdsourcing service looks from the worker's side, for example, by registering and trying to do some tasks yourself or by reading forums for workers.

Post reply on HN