Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% of Google's Emotions Dataset Is Mislabeled

surgehq.ai

Post reply on HN
Print view View original

101–110 of 146 posts

Re: 30% of Google's Emotions Dataset Is Mislabeled

#101

Post by drewcoo »

EricMausler wrote: post #89
alxlaz wrote: post #57

Earlier quoted context omitted.

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent. The one time I needed anything remotely like this I just took the old "nobody said programming was…

But then STEM would have to admit that English Lit Majors didn't waste their money /s

English Lit Majors can't find work? Try MTurk!

Re: 30% of Google's Emotions Dataset Is Mislabeled

#102

Post by amelius »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options?

Use the consensus of 3 or more annotators (or median).

Re: 30% of Google's Emotions Dataset Is Mislabeled

#103

Post by zharknado »

david_draco wrote: post #44
pmontra wrote: post #32

Earlier quoted context omitted.

US English != Indian English especially if you have to actually know the cultural details behind some sentences. I bet US native speakers would have similar failure rates at labeling English sentences from Indian media, because they belong to different cultures.

Is the "right answer" a classification based on US English with background knowledge of US cultural background, or is the goal to build a global sentiment data set? You and OP ("the indians labelers don't know how to do it correctly") seem to want the former, so it would be good to state that goal upfront.

I think it’s valid to question US-centrism broadly in datasets like this, but presumably that choice was made in selecting the Reddit sample, which is dominated by North American English.

A model is more useful if it approximates the speaker’s intent, not the listener’s interpretation. “Right” is complicated in language, but it’s hard to see how you’d use a model full of cross-cultural misunderstandings.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#104

Post by drewcoo »

sireat wrote: post #80
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc. All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst. "Jia…

There are ways to detect problems with results due to things like fatigue or attenuation. And there are ways to deal with those problems (like forced break time and switching up tasks).

It's a wonder that none of that is built in. But maybe things like MTurk aren't built to maximize worker effectiveness because it costs too much. Are there better quality crowd-sourcing options?

Re: 30% of Google's Emotions Dataset Is Mislabeled

#105

Post by LanceH »

nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

In my first few months as programmer in the 90's, I realized that human inputs were sketchy. It was from a form field for which state someone was from (USA). Fifty possible states, 2500 different entries. Sure there was a bit of garbage, but 95+% were recognizable states...but why random capital letters? How do you get a space in Wyoming?

It was a good lesson at the start of my career that I see playing out over and over. When I see some cool demo of statistics across the country or globe, I'm more impressed by the effort of cleaning the data than the stats behind it.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#106

Post by kortex »

> Hi dying, I'm dad! – mislabeled as NEUTRAL, likely because labelers don’t understand dad jokes

In their defense, what could be more True Neutral alignment than dad jokes? Nothing to gain but the quiet enjoyment of making the room groan and roll their eyes.

Really though, the issue here is context, but also the complexity of human communication. The sensitivity and tone highly depends on the situation. Clearly the preceding moment is someone stating "I'm dying". But that itself is contextual. Are they literally facing mortality, merely inconvenienced and being hyperbolic, or laughing? If the former, is "Hi Dying, I'm Dad" being glib, to soften the blow of a dire confession, or being highly insensitive and poking fun in a serious moment? Is it in the context of a longer joke, which subverts the meanings yet again?

A lot of these comments are worse than useless without context. Reddit really likes improv-banter style humor in comment chains. One comment builds on another builds on another, all referencing in-jokes, and usually slathered in sarcasm.

Honestly Reddit comments are probably one of the worst sources to try to build a sentiment model from, from an engineering perspective.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#107

Post by dwringer »

alx__ wrote: post #38

Language is hard! Even I, a seasoned native internet dork, have trouble knowing if someone's comment is sarcasm, irony, or something in between. Also, new phrases emerge all the time that turn a phrase on its head, and it has a different emotion. How many feelings can you evoke with a simple, FUCK!

Further compounded by the fact that often if I say something and get asked, "was that sarcasm, irony, or something in between?" the best I can probably do is "Eh, more or less".

Re: 30% of Google's Emotions Dataset Is Mislabeled

#108

Post by kortex »

rkagerer wrote: post #6
scottlawson wrote: post #3

this is a genuinely great read. Author does a great job providing examples where context is critical, and explains how the dataset not only has labeling errors, an even deeper problem is how it models language in general. Since words only have meaning within a context, your model should reflect that somehow. What wasn't really explored I'm this article was to what quantitative degree context sensitivity matters. The…

Context is key. My dog has a better understanding of context than any AI I've ever met. (I mean that sincerely - since becoming a pet owner it's something I've really marveled at).

Seriously. Half the time, I feel like the command I'm giving is just being interpreted as "do the needful", and she just figures it out, based mostly on tone, context, and memory.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#109

Post by drewcoo »

tqkxzugoaupvwqr wrote: post #84
alxlaz wrote: post #81

Earlier quoted context omitted.

Surely if it's such an easy job, you can fit it along your other tasks. If you can't then perhaps it's worth re-examining if it's as easy as you think it is.

It’s easy but requires time. Like turning a page is easy but turning a million pages takes time.

Design a machine, possibly including humans as "parts," to automatically turn those million pages. Maybe the machine is just a single human, serially, turning pages, but that's slow. There must be something better!

How easy is it to design a reliable system, considering human/machine interaction and everything we know about human behavior and constraints like human attention, potential injury, dry fingers, etc.?

How simple is the design?

Re: 30% of Google's Emotions Dataset Is Mislabeled

#110

Post by wongarsu »

amelius wrote: post #102
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Use the consensus of 3 or more annotators (or median).

But is paying 3x $5/h getting better results than hiring a local collage student for $15/h?
Post reply on HN