Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

30% of Google's Emotions Dataset Is Mislabeled

surgehq.ai

Post reply on HN
Print view View original

111–120 of 146 posts

Re: 30% of Google's Emotions Dataset Is Mislabeled

#111

Post by stjohnswarts »

MonkeyMalarky wrote: post #93
sireat wrote: post #80

Earlier quoted context omitted.

I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc. All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst. "Jia…

I'm eager to see what nefarious things Google will do once they've mastered the art of identifying all the busses in a photo!

their motorcycle, hills, signal light, and traffic light AI will break the world when it's over!

Re: 30% of Google's Emotions Dataset Is Mislabeled

#112

Post by sjtindell »

kortex wrote: post #106

> Hi dying, I'm dad! – mislabeled as NEUTRAL, likely because labelers don’t understand dad jokes In their defense, what could be more True Neutral alignment than dad jokes? Nothing to gain but the quiet enjoyment of making the room groan and roll their eyes. Really though, the issue here is context, but also the complexity of human communication. The sensitivity and tone highly depends on the situation. Clearly the p…

True, we’re trying to produce bots that reliably do things (make people laugh) that humans can’t even do reliably. People who can feel out a room and use the right joke, or the right reassurance or whatever, are not even very common.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#113

Post by simiones »

sireat wrote: post #80
stingraycharles wrote: post #7

Earlier quoted context omitted.

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

I was using MTurk for labeling about 10 years ago. To see the other side I also did a 1 Month stint as a MTurk worker, earning about $300. It is absolutely horrible work, and I used MTurk subreddit to find the "decent" jobs. I had the special firefox extension which ranked the job givers etc. All jobs were below 1st world minimum wage and were incredibly depressing. I think the adult content ones were the worst. "Jia…

> Compare it to newest Google reCaptcha - you know those labels are going to be used for evil at some point in the future.

I for one always try at least once to get something wrong, often several times if it doesn't go through immediately, depending on urgency of my task. I hate being made to work for someone else like this.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#114

Post by stjohnswarts »

codedokode wrote: post #55
bragr wrote: post #46

Earlier quoted context omitted.

>I honestly believe most MTurkers just clicked random BS in order to complete the tasks as soon as possible. This is correct based on my (limited) mechanical turk experience. Most tasks pay peanuts (minimum payout can be as low as $0.01) so the only reasonable way to make an income is to complete as many tasks as humanly possible, and doing anything but clicking random buttons would slow them down. I doubt paying mor…

You can filter out bad workers by preparing an additional well-labeled dataset and removing those who made even a single mistake in it. Also you can give the same task to several workers and check if they give the same label. However this won't protect against a bot using multiple accounts and giving answers based on a hash of a question so that the same question gets the same answer in every account.

"removing those who made even a single mistake" means you don't want humans working on this lol. Humans will always make mistakes.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#115

Post by mdorazio »

stingraycharles wrote: post #7
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…

> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options?

Fwiw, in a similar situation I did about half myself and farmed out the other half to my retired parents. Using trusted people was the only way I found I could get high accuracy without spending thousands of dollars. But, as I frequently tell people, the hard part of ML isn’t the model/code, it’s the training data.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#116

Post by nostrademons »

Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment label. That sets a theoretical limit on the effectiveness of machine-learning algorithms, because if humans can't agree, then any product that needs to take an opinion is going to be wrong 30% of the time. (This is probably also why Big Tech companies are leaning so heavily into personalization.)

Also reminds me of when I asked a veteran therapist what the most surprising part of his job was, and it was:

1.) The variety of ways that different people perceive a given situation, and just how much neurodiversity is out there.

2.) How everybody expects that everyone else will see the situation exactly the same way they do.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#117

Post by stjohnswarts »

andy_ppp wrote: post #98
nmfisher wrote: post #4

Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…

Why can’t you do the dataset several times with different labellers, building up a more statistical probability for a label than a pure guarantee. It’s worth noting that different cultures have different interpretations of emotions too (famously Russians don’t smile much even though they are extremely helpful in my experience, I’m still not quite sure what the head wobble in India actually means let alone some of the…

Or pay better like other people are suggesting. If you have to "average over" 3 data sets why not just pay $15 instead of $5 and save the computation if $15 or so seemed to be the threshold for getting humans to be good data labelers?

Re: 30% of Google's Emotions Dataset Is Mislabeled

#118

Post by wongarsu »

malikolivier wrote: post #62

I assumed I was quite fluent in English, even in slangs, having seen a fair share of both American and British movies. Now that I see the examples given, I think I would have mislabeled most of them too, even if I were highly motivated to label them. Though it's normal for any language, it's very interesting how English is variable between dialects and time periods when it comes to slang. There are so many regional s…

Movies don't use the language mislabeled here. Youtube and Twitch do, sometimes excessively so.

I'm pretty sure you would be able to find half of the "mislabeled as negative" sentences shown in a single 15-minute "Among Us" video.

There are others I would have mislabeled too. I think it shows how you need a grasp of the subculture the comment is coming from to reliably label it. Most 15-year olds would probably ace those examples, no cap.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#119

Post by jasonlotito »

The author "previously led AI, Data Science, and Human Computation orgs at Google, Facebook, and Twitter."

And is now writing an ad critical of the company he worked at, about an area he was involved in leading.

This is an interesting route to take in a career. Work for a company, make mistakes, move to another company, and use your old mistakes as a selling point for the new company.

I know this is a harsh take, but it doesn't instill any confidence in the results here. What happens when mistakes happen at Surge? Are the people who made the mistakes going to be around to fix them, or are they going to jet off to another position where they once again talk about their previous failures.

Re: 30% of Google's Emotions Dataset Is Mislabeled

#120

Post by wongarsu »

nostrademons wrote: post #116

Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment label. That sets a theoretical limit on the effectiveness of machine-learning algorithms, because if humans can't agree, then any product that needs to take an opinion is going to be wrong 30% of the time. (This is probably also why Big Tech companies are leaning…

That sets the theoretical limit for an algorithm trained on a dataset labeled by outsiders. Most people should be able to label the sentiment of their own statements with much higher accuracy.

Making such a dataset is much harder than letting Mechanical Turk workers label reddit comments, and you somehow have to set up a situation where people are honest about their labels, but the rewards might be worth it.

Post reply on HN