I don't understand why they're doing this on the Yelp/IMDB datasets. Here's the paper they're citing that is doing the same thing with gated neural nets:
http://www.emnlp2015.org/proceedings/EMNLP/pdf/EMNLP167.pdfHowever that paper also has the polar score which is a 2-class classifier - 1 star + 2 star vs 3 star + 4 star.
Basically, the rating scores are clearly ordinal, but they're completely disregarding that information as if a 4 is a completely different star review than a 5 in a yelp restaurant review or a 9/10 is different from 8/10 for IMDB review.
More useful estimate of success would be how close they got to real rating and possibly penalize disproportionately when a model gets the rating completely wrong. Something like mean squared error of the final score vs predicted score.
I understand that these are just used to benchmark the algorithms against each other, but why not use something more relevant like a topic classification? That is a real n-classes problem.
General disregard for domain-specific information is very prevalent in machine learning papers and it makes real world applications difficult because real world evaluation metrics are different so classifiers that are marked as inferior in such evaluation might actually be better.
In both of their evaluations, why not try
F(y) = exp(yj) / sum(exp(yi) for i in range(1, n+1))
p1 = F(y1)
pj = F(yj) - F(yj-1), for j ≥ 2
for softmax? In other words, subtract cum prob of previous category.
Or use kappa function?
At least the naive bag of words comparison classifier should have used ordinal logistic regression instead of n-class logistic classifier.