Why Ratings Systems Don't Work

> The problem with any commercial website with ratings (Amazon, Yelp, etc., etc.) is that there is HUGE incentive to game the ratings.

While I agree that this is a problem, I think a bigger problem is a simple matter of scale:

Amazon is huge, and many people buy things, but they don't split reviews and ratings by what kind of person is rating them. If they wanted to make an improvement, why not show me only ratings and reviews by people who are similar to me? They have tons of data about me and other people who use the service, so it should be possible for them to say "people like you rated this on average a 4, but everyone in the world rates it an average of 2.5."

That's much easier than having to read all the reviews and decide if the person is in my demographic or whether I agree with their review.

Ratings leave something to be desired for sure. You can tell if something is worth checking out if it's highly rated but other than that they aren't tremendously useful.

That being said I'm not a fan of the goodfil.ms graph at all. Why do I care about how rewatchable someone else thinks a movie is? Do I want to watch it again? I would know because I'd have seen it.

I think rewatchable is the wrong term. Fun is what you're looking for. Major Payne is a fun movie. It is incredibly rewatchable but I don't need a website to tell me that. I could use a website to point me to it if I haven't seen it before.

I think the site is on the fringe of something important though. There are multiple ways to rate a movie, and mood plays a huge role in what you want to watch at any given moment.

If something has 1000 ratings but is only 3.5 stars whereas something with 1 rating and 5 stars, what does that say about the average? Not much. It just says there's more people to rate something and have an opinion about it. Getting people to rate something is difficult in and of itself and only the people who are on the love/hate spectrum will rate something.

I like the histograms as it reveals a little into the rating. After all, if a single person gave it 1 star and everyone else rated it 3 or more, the average is likely skewed because of that one person who is clearly "gaming" the system because they weren't happy.

I think ratings should take into account intent. If multiple people are rating it 1 star, then clearly it should be weighted downward. However, if a single person out of 100 people gave it 1 star, I don't think the average should be weighted evenly. It's a difficult problem to solve and XKCD is just making a joke.

Average rating as a measure of "goodness" is wrought with statistical problems. Without the context of other statistical modes, looking at mean is pretty useless. However, people don't want to look at summary statistics for each item (mean, median, mode, std/var, skew, etc). So we try to come up with scalar metrics that capture "goodness" or "coolness" or whatever. Popularity (how ever you define it) is a common one to use. Here's a good comparison of popularity models: http://blog.linkibol.com/2010/05/07/how-to-build-a-popularit.... In the past I've been pretty happy with "Bayesian average" - it's simple to implement and gives good results.

But if you really want to dig into it, you have to consider all kinds of stuff like bimodal distribution of ratings (controversial items), rater quality/consistency, age or ratings, etc, etc.

It's really not as simple as you'd think!

In summary then, using the mean value to summarise data from an unknown distribution can be problematic. Not exactly ground breaking.