Why Ratings Systems Don't Work

You should consider throwing up some histogram pairs of the data shown on the scatter plots.

It looks like they would still give a good picture of the data, as the trends are mostly linear.

I'll meet you halfway. It's a terrible comment about a terrible article. I agree that it's laudable to try to invent an improved version of online ratings, but this article wasn't effective in convicing me that they've succeeded.

Their argument that historgrams are just.awful. (I didn't care for the extra periods) seems to have two components: 1. asserting that historgrams are bad 2. showing us 3 histograms and saying they tell you the same thing about all 3 movies, when in fact there is a very clear and important difference between the histograms.

Its completely obvious from inspection ("people are really good at seeing patterns") that Starship Troopers has a much lower percentage of 5 star ratings than the other two, and a much higher fraction of 0 star ratings. It also appears to me that the Fifth Element has a higher fraction of 4 or 5 star ratings, and is probably the most apprciated of the 3 films, although Blade Runner is fairly close.

If you are going to cherry a set of 3 specific films to make your point, you should be sure to at least pick 3 films that support your point instead of refuting it.

We then learn about their hypothesis that while 5-star rating system sucks, a system that relies on two correlated 5-star ratings is great. They demonstrate this by using the two question system to draw the exact same conclusion I drew from the histograms of the 1 question ratings.

I would've liked some sort of objective attempt to compare the two rating systems. Perhaps it would be possible to measure how frequently the two question system leads people to make a better choice than the one question system, or at least some sort of statistical wonkery that would purport to show me that the two question system in practice draws more distinctions than the one question system. Unfortunatley we only get this one rather uninspired example ("watch this if you’re in the mood for something really good").

They also didn't address why "would you re-watch this film" is a better choice than any other second question. There are attempts to justify it being a good question, but no real evidence that other questions were tried and didn't perform.

Finally, the thing that really irked me was that this proposed system doesn't seem to do anything to address most of the actual problems with the regular 5 star system, namely that people who feel really strongly about something are more likely to rate so that most ratings tend towards the extremes, and that without context we have no idea why someone rated something a 5 instead of a 1. Those problems now exist along two dimensions instead of one.

I see this less as an article and more as an advertisement hitching a ride on an xkcd comic.

Why don't rating systems just give a simple yes/no question to the reader. In the case of rating a movie, just ask "Would you watch this again?" or in the case of purchasing a product from Amazon, "Would you purchase this item again?"

I'd rather a boolean system than one where someone's 4-star rating is different than my 4-star rating. Whenever I see a multi-star rating system, I remember back to a prof I once had that said "The top grade is B+. A's are reserved for God." Albeit disgusting, it taught me that everyone has a different rating scale.

The problem is people do not take the time to rate products (movies in this case) unless they are notably good or notably bad. I believe Rotten Tomatoes has the most realistically accurate system for rating movies. Either you think someone should see the movie, or shouldn't. If the majority of people are on the fence, then it will get a rating around 50%.

I'm not sure why I would care about the "rewatchable" metric. I only look at reviews for a movie I haven't seen. If I've seen it I can decide myself if I'd enjoy watching again. So subtracting that you are left with the normal star system rating.