I've worked on project with more difficult labeling, and we were able to get fairly accurate results. There are tons of standard practices that produce better results, so why did Google ignore them.
30% of Google's Emotions Dataset Is Mislabeled
31–40 of 146 posts
Re: 30% of Google's Emotions Dataset Is Mislabeled
#32Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…
>Farm it out to low-paid non-native speakers The paper claims: >“All raters are native English speakers from India.”
US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.
Re: 30% of Google's Emotions Dataset Is Mislabeled
#33Earlier quoted context omitted.
>Farm it out to low-paid non-native speakers The paper claims: >“All raters are native English speakers from India.”
US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.
The blog post highlights this specific point - "US English" and "Indian English" really aren't the same English (in fact, I'd probably go even further and state that "Reddit English" and "US English" probably aren't the same English either).
Likewise, the Common Voice English dataset isn't great for ASR training outside India, either. There's a huge proportion of Indian speakers, and their data doesn't really help train ASR systems for non-Indian accents.
Re: 30% of Google's Emotions Dataset Is Mislabeled
#34> let’s look at the labeling methodology described in the paper. To quote Section 3.3: > “Reddit comments were presented [to labelers] with no additional metadata (such as the author or subreddit).” > “All raters are native English speakers from India.” This does not look good even on paper. No wonder the errors were abundant Also a labeling system that has no entry for sarcasm is totally going to work guys!!1 /s
Isnt knowing the subreddit valuable information to determine sentiment?
correct, the article has some good examples about this
But even without that, the context or just the topic of the discussion should help
Re: 30% of Google's Emotions Dataset Is Mislabeled
#35You get what you paid for!
Re: 30% of Google's Emotions Dataset Is Mislabeled
#36Re: 30% of Google's Emotions Dataset Is Mislabeled
#37> let’s look at the labeling methodology described in the paper. To quote Section 3.3: > “Reddit comments were presented [to labelers] with no additional metadata (such as the author or subreddit).” > “All raters are native English speakers from India.” This does not look good even on paper. No wonder the errors were abundant Also a labeling system that has no entry for sarcasm is totally going to work guys!!1 /s
Isnt knowing the subreddit valuable information to determine sentiment?
Yep. And familiarity with Reddit/subreddit memes and inside jokes. And entire subreddits devoted to parodying styles of comment which are utterly earnest on other reddits which I'm not sure how you'd even begin to classify...
Re: 30% of Google's Emotions Dataset Is Mislabeled
#38Language is hard! Even I, a seasoned native internet dork, have trouble knowing if someone's comment is sarcasm, irony, or something in between. Also, new phrases emerge all the time that turn a phrase on its head, and it has a different emotion.
How many feelings can you evoke with a simple, FUCK!
Re: 30% of Google's Emotions Dataset Is Mislabeled
#39Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…
Can a human validate a label with less effort than it took to create it? Or maybe validating statistically is enough?
Depends what it is. I've had reasonable success with "validating" ASR transcripts by loading up the annotator's transcript, running the audio at 2x speed and just clicking "yes" or "no" to keep the good ones and bin the bad ones. It's roughly 5x faster to do this than to come up with transcripts from scratch, so if the annotators are 5x cheaper, then you come out ahead. You can go even further and pre-filter labels to discard any where inter-annotator agreement falls below some threshold (i.e. 3 people label the same piece of data, and you only include a sample when at least 2 annotators give the same label). You can also use that to discard all annotations from annotators who regularly disagree with the majority.
This is just the reality of outsourced data labelling. One thing I think is really important is to structure the compensation well, so that labellers get paid more when they do a better job. Paying per sample is a terrible idea, and even I was guilty of this - back in university I was paid $20 or so to hand-write 500 words on a resistive touchscreen to train a handwriting recognition model. I won't say I half-assed it, but I remember trying to get through it as as quickly as possible to get my money and go for beer (I think I also justified it to myself on the basis that sloppy samples would help make the bounds of the dataset distribution more robust!).
Re: 30% of Google's Emotions Dataset Is Mislabeled
#40> LETS FUCKING GOOOOO Could be either anger (let’s fight) or enthusiasm (let’s do it!). Hard problem.
I don't see anger in "LETS F**ING GOOOOO". It's just a comment that says "let's do X" in impatient and enthusiastic manner.
How about, "LETS F*ING GOOOOO YOU DINGBAT", now if that's a comment between friends the addition of the insult might be said in jest and still be impatient/enthusiastic, or does by adding the insult to the end automatically label it as combative?
I realise this wasn't part of the dataset, more making a point that written language without context ( and sometimes even with ) is subject to huge amounts of reader interpretation.