81–90 of 146 posts
Earlier quoted context omitted.
If it's such an easy job, why outsource it instead of doing it yourself?
Because there’s other stuff you need to do that you can’t outsource.
If you can't then perhaps it's worth re-examining if it's as easy as you think it is.
I have three questions now: * How much (per comment) are these "native speakers from India" paid? * How many comments do they have to label in an hour (or in a minute)? I guess it's more than 2 comments in a minute. * What if the comment is sarcastic and this can only be understood from its context?
Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…
Earlier quoted context omitted.
Because there’s other stuff you need to do that you can’t outsource.
Surely if it's such an easy job, you can fit it along your other tasks. If you can't then perhaps it's worth re-examining if it's as easy as you think it is.
Anyone who's dealt with any kind of human-annotated datasets would be familiar with these kind of errors. It's hard enough to get good clean labels from motivated, native-English speaking annotators. Farm it out to low-paid non-native speakers, and these kind of issues are inevitable. Annotation isn't a low-skill/low-cost exercise. It needs serious commitment and attention to detail, and ideally it's not something yo…
For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…
But it seems they’ve been acquired a few years ago, so I have no idea if the quality is still the same.
this is a genuinely great read. Author does a great job providing examples where context is critical, and explains how the dataset not only has labeling errors, an even deeper problem is how it models language in general. Since words only have meaning within a context, your model should reflect that somehow. What wasn't really explored I'm this article was to what quantitative degree context sensitivity matters. The…
As an end user I face similar problems in UI translations: A lot of failed translations are made on context deprived text, on a false notion that additional contexts are only required in nuanced edge cases. In reality it is almost always necessary especially in UI texts where words are more loaded and supplemented by visual indications.
“Context” as often said might need to be defined with more depth. As it stands it is used as almost a post-hoc explanation as to why a particular output of an arbitrary language-related tasks is considered incorrect and should be discarded.
Earlier quoted context omitted.
I'd love to chat. Want to reach out to the email in my profile? I'm the founder of a startup solving this exact problem ( https://www.surgehq.ai ), and previously built the human computation platforms at a couple FAANGs (precisely because this was a huge issue I always faced internally). We work with a lot of the top AI/NLP companies and research labs, and do both the "typical" data labeling work (sentiment analysis,…
Good news for you: being your target audience, we actually did have you guys on our radar.I For the scale of our project, however, the price point was prohibitive. We ended up building a small cli tool that interactively trained the model, and allowed us to focus on the most important messages (eg those where positive/negative sentiment was closest, the labels with the smallest volume, etc). EDIT: If I now look at yo…
If you have more time than money it might not make sense, but at that price point I could save myself a lot of time by just working a few extra hours doing SE and let someone else do 3x that amount of labelling.
Earlier quoted context omitted.
How about, "LETS F*ING GOOOOO YOU DINGBAT", now if that's a comment between friends the addition of the insult might be said in jest and still be impatient/enthusiastic, or does by adding the insult to the end automatically label it as combative? I realise this wasn't part of the dataset, more making a point that written language without context ( and sometimes even with ) is subject to huge amounts of reader interpr…
This one is indeed hard to label. PS Hope they will never begin solving philosophy problems with ML.
Earlier quoted context omitted.
For a sentiment / emotion classification project, we (2 founders) just ended up doing most of the labeling ourselves. It was a big grind, but given how abysmal the performance of “crowd-sourced” solutions are (eg Amazon Mechanical Turk), and how incredibly important the quality of these labels are for training a model, it made the most sense. I wonder how others do this kind of thing. Assuming I have 100k text blurbs…
> I wonder how others do this kind of thing. Assuming I have 100k text blurbs I want to classify of sentiment and/or emotion and want at least 95% accuracy, what are the options? Others who successfully do it have exactly the same secret sauce that you do: they assign it to someone who is well-compensated and competent. The one time I needed anything remotely like this I just took the old "nobody said programming was…
Earlier quoted context omitted.
Surely if it's such an easy job, you can fit it along your other tasks. If you can't then perhaps it's worth re-examining if it's as easy as you think it is.
It’s easy but requires time. Like turning a page is easy but turning a million pages takes time.
It's attractive to think that this is just like classifying one data point over and over again. It's nothing like that, just like crossing the Atlantic from Galway to New York is nothing like kayaking around Mutton Island over and over again.