How Not To Sort By Average Rating
evanmiller.org
How Not To Sort By Average Rating
1–10 of 34 posts
Re: How Not To Sort By Average Rating
#2I recently needed to do this for sorting search results. My approach was to calculate a Bayesian average (http://en.wikipedia.org/wiki/Bayesian_average ), which was also pretty simple (once I found a formula to copy from here: http://www.thebroth.com/blog/118/bayesian-rating )
It seems to me that the Bayesian average is superior, because it considers the value relative to other items.
Re: How Not To Sort By Average Rating
#3Oh god, wall of math!
/me hangs head in shame.
Re: How Not To Sort By Average Rating
#4Nice. I'd figured out the problems with the 2 wrong solutions myself but I wouldn't have known where to start looking for the better approach.
Re: How Not To Sort By Average Rating
#5The wrong solutions the article describes are certainly wrong. It doesn't really make much of a case that the particular solution it proposes is particularly good.
For instance, here's another much simpler formula that avoids the problems described in the article: score = (pos+1)/(pos+neg+2). It's the posterior mean of Pr(random person likes the product), if your prior is uniform on [0,1]. You can adjust this to favour items with more ratings by, e.g., using a prior that "prefers" smaller values of that probability; one easy way to do that is to pretend that every item starts with a certain number of positive reviews and a certain number of negative reviews; you end up with a formula of the form (pos+A)/(pos+neg+B).
Is the article's formula better than this? Probably, but the author hasn't said why.
Re: How Not To Sort By Average Rating
#6This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of reviews we have at RateItAll. In general, with a system like this, the items with the most reviews are going to rise to the top. This might not be what you want. You can easily see this effect with something like our "Events of 2010" list: http://www.rateitall.com/t-2987869-events-of-2010.aspx
Re: How Not To Sort By Average Rating
#7This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of reviews we have at RateItAll. In general, with a system like this, the items with the most reviews are going to rise to the top. This might not be what you want. You can easily see this effect with something like our "Events of 2010" list: http://www.rateitall.com…
>items with the most reviews are going to rise to the top
Yes and no. Anything with a larger N is going to make for greater confidence in the rating. But that rating still depends on the number of positive and negative reviews. So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. The algorithm would presumably be very confident about placing it in the middle of the pack.
Re: How Not To Sort By Average Rating
#8We have used this algorithm in our side project http://movieorwhat.com/month/ - it seems to be working very well!
Re: How Not To Sort By Average Rating
#9I recently needed to do this for sorting search results. My approach was to calculate a Bayesian average ( http://en.wikipedia.org/wiki/Bayesian_average ), which was also pretty simple (once I found a formula to copy from here: http://www.thebroth.com/blog/118/bayesian-rating ) It seems to me that the Bayesian average is superior, because it considers the value relative to other items.
The Bayesian example given in the Wikipedia article looks like it works best with interval level data. Not sure it would work as well with ordinal data, because ordinals don't suffer to the same degree from outliers.
The second example also isn't ordinal data—it's just nominal "liked" vs "didn't like."
Both examples are interesting, but don't necessarily get at the problem with ordinal data introduced by the article.
Re: How Not To Sort By Average Rating
#10reddit uses the Wilson algo for comment sorting:
http://blog.reddit.com/2009/10/reddits-new-comment-sorting-s...