here is what I don't understand about deep NLP (please keep in mind that I just began exploring this field): I am currently working on an algorithm that uses elementary text cues in combination with large data-table lookups to determine things like relevant keywords of news articles scraped from various sites. I have given my results to hundreds of people independently to provide me with some feedback regarding the q…
Certainly in many cases a better accuracy can be both reasonably needed and reasonably possible (i.e. if humans can do it, then it's obviously possible).
One measure that is used, and is a bit similar (though with a major difference) is "inter-annotator agreement", i.e., you ask the same question to multiple people and note how often they agree. That would be considered a reasonable ceiling, is a measure of how objective/subjective the question is, a measure of how often there really is a single "correct answer"; for some problems that metric is near 100% and can be reasonably beaten by a good system, because the mismatches are caused by human mistakes instead of true disagreements; for others (e.g. some forms of emotion/sentiment/sarcasm analysis) 80% is unreasonably good, since the text doesn't have enough information to decide for sure.
Also, an answer to (3) is that to get a state of art result (as opposed to a simple baseline) with non-DNN methods you need a quite complex system and lots of custom feature engineering. If you have (or get) one, that's not an issue, but if developing a system from scratch, a good DNN system needs less labor than a good "classic" system. For example, a major point in neural machine translation is that it not only gets better results, but that it can get them with a much simpler NLP pipeline. When a "classic" system needs to integrate 10-30 additional separate modules (ML or with manually crafted rules) for handling various types of special cases or feature analysis, much of that (though not all) can be learned by a deep neural network directly in end to end training; so if you go directly to DNN then you avoid the (huge!) work of implementing them manually.