>
Criteria for "the best": Making such a recommendation it’s a lot like saying “this is the best place to look for a movie rating,” which is an evaluative statement, resting on some criteria used to determine what is better, what is worse or worst, and what is best, in this case. For my recommendation, I will use one single criterion: a normal distribution.I don't get it. When evaluating which ratings system is better, why should we expect review ratings to follow a normal distribution (nevermind why it's plausible any "single criterion" is suitable for the task)? I appreciate the lengthy analysis but it seems like a bunch of scientific mental masturbation when the premise is so arbitrary and possibly fallacious.
The data sample consists of "the most voted and reviewed movies in 2016 and 2017". Almost certainly this sample is skewed towards the most heavily advertised and publicized movies -- whether it's during the theatrical run, or in the Oscars/awards season push movies.
I won't argue that "more money == better", but movies with a heavy advertising budget are likely to have a high production budget and/or more well-known actors/directors/etc. Assuming that over the decades, movie studios have evolved to know what audiences like, we can expect that moviegoer ratings skew towards the positive, because producers and studios who put out more $100M fuckups than successes tend to not survive.
So IMDB's and Fandango's skew towards the positive seems expected, assuming that they represent the common movie goer. Fandango's positive skew seems especially expected because these are ratings from people who ostensibly have paid $10-$20 to see the movie. A positive skew can come from the fact that these people put their money where their mouth is -- they wouldn't willingly throw money at a bad movie. Or you can argue that there's some economical psychology at play, that they've fooled themselves into thinking that a 3-star movie is a 4-star movie because they just spent $20 on it.
Of course, it's not just whether or not the statistical skew is expected, but whether it is good for you. Well, that depends on you. If you're an average moviegoer, then IMDB/Fandango may be the perfect scale for you, as both scales are more democratic (each vote is equal, ostensibly), and higher-rated movies means more users, well, gave them higher ratings. From a cynical perspective, this means that you're just one of the masses who like what the movie studios know you like, but that doesn't mean that IMDB/Fandango are inferior scales for you.
Likewise, we shouldn't expect the Tomatometer, if it represents published critics, to be follow a normal distribution. Again, we have no reason to think that the most rated movies on IMDB will fit a normal distribution of quality. And we have even less reason to believe that the critics, whitelisted by RT, have a normal distribution of ratings for movies that are mass-marketed.
The author says he feels weird about RT because it's an aggregation of a binary judgment of critic reviews (e.g. ratio of reviews judged to be positive vs. negative):
> Given the last consideration and the shape of the distribution, the tomatometer doesn’t meet my criterion. It could be that a larger sample would do it more justice, but even so, if I were to recommend it, I would do it with some reserves because of the vague positive or negative rating system.
That's the rating system that Steam and Netflix use. Of course, Netflix's rating is personalized, so it's not just a straight up aggregation, but the fact is that they trust it's granular enough for user data collection. The author doesn't give a good reason why we should find RT's system vague, nevermind why the distribution of RT ratings should be normal given the selection bias of the sample.