121–130 of 146 posts
Write an emotion that is expressed in a given image label.
Label: "[label]"
Emotion: [filled by GPT3]
Then, for "you almost blew my fucking mind there." -> "Suprise", for "hell yeah my brother" -> "Pride", "Nobody has the money to. What a joke" -> "Anger". Though, to be fair, for "Yay, cold McDonald's. My favorite." it was "Happiness". Still better than the crowdsourced human baseline.
Anger
Earlier quoted context omitted.
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Use the consensus of 3 or more annotators (or median).
But is paying 3x $5/h getting better results than hiring a local collage student for $15/h?
Answer is yes, because those $5/h workers are likely just as educated, but from a less affluent part of the world.
Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…
>Farm it out to low-paid non-native speakers The paper claims: >“All raters are native English speakers from India.”
Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment label. That sets a theoretical limit on the effectiveness of machine-learning algorithms, because if humans can't agree, then any product that needs to take an opinion is going to be wrong 30% of the time. (This is probably also why Big Tech companies are leaning…
That sets the theoretical limit for an algorithm trained on a dataset labeled by outsiders. Most people should be able to label the sentiment of their own statements with much higher accuracy. Making such a dataset is much harder than letting Mechanical Turk workers label reddit comments, and you somehow have to set up a situation where people are honest about their labels, but the rewards might be worth it.
> Hi dying, I'm dad! – mislabeled as NEUTRAL, likely because labelers don’t understand dad jokes In their defense, what could be more True Neutral alignment than dad jokes? Nothing to gain but the quiet enjoyment of making the room groan and roll their eyes. Really though, the issue here is context, but also the complexity of human communication. The sensitivity and tone highly depends on the situation. Clearly the p…
True, we’re trying to produce bots that reliably do things (make people laugh) that humans can’t even do reliably. People who can feel out a room and use the right joke, or the right reassurance or whatever, are not even very common.
I think it's still valuable to be able to detect when something is joke/satire/sarcasm/irony/slang, especially in the context of content moderation, because quite often it totally flips the sentiment valence. A perfect example is "I'm literally dying" - "literally" meaning in the exact or truest sense, "dying" meaning sloughing off the mortal coil (very bad)- vs "literally" meaning "figuratively, but in an extreme sense" and "dying" from laughter (very good).
Reminds me of the stat I heard that humans are only 70% accurate at sentiment analysis, because different people will not agree on the appropriate sentiment label. That sets a theoretical limit on the effectiveness of machine-learning algorithms, because if humans can't agree, then any product that needs to take an opinion is going to be wrong 30% of the time. (This is probably also why Big Tech companies are leaning…
Only tangentially related, is your username a pun on "Nostradamus", "Nostril", and "Nasal Demons"? If so, that is very witty!
Earlier quoted context omitted.
That sets the theoretical limit for an algorithm trained on a dataset labeled by outsiders. Most people should be able to label the sentiment of their own statements with much higher accuracy. Making such a dataset is much harder than letting Mechanical Turk workers label reddit comments, and you somehow have to set up a situation where people are honest about their labels, but the rewards might be worth it.
That's the personalization angle. A lot of effort's being expended on transfer learning + training at the edge, where you start with a general model trained on humanity and then it gradually learns about the specific human(s) it's interacting with.
Personalizing to the actual subculture the interaction takes place in would be another step. You probably need both.
Indian English is just as valid as American English. The problem here is they used Indian English speakers to rate Reddit comments, most of which are using American English idioms.