Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How Not To Sort By Average Rating

evanmiller.org

Post reply on HN
Print view View original

11–20 of 34 posts

Re: How Not To Sort By Average Rating

#11

Post by Automatt-2 »

Perceval wrote: post #7
Automatt-2 wrote: post #6

This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of reviews we have at RateItAll. In general, with a system like this, the items with the most reviews are going to rise to the top. This might not be what you want. You can easily see this effect with something like our "Events of 2010" list: http://www.rateitall.com…

>items with the most reviews are going to rise to the top Yes and no. Anything with a larger N is going to make for greater confidence in the rating. But that rating still depends on the number of positive and negative reviews. So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews. The algorithm would pres…

>So if a product has a large number of ratings, but they are 50/50 positive and negative, then it's not going to rise to the top simply because it has the most reviews.

What I'm saying is that it will rise to the top because even though only 50% of the ratings are positive, it will have many many more positive ratings than the nearest next item. That 2nd item may have a much higher percentage of positive ratings, but a much lower number of total ratings.

Re: How Not To Sort By Average Rating

#12

Post by JustSemantics »

It's very nice to see a mathematically motivated piece on how to sort based on users ratings. But there are two problems with the solution presented, the first minor, and the second more serious.

Minor: Distributions of ratings are not usually normal about the average possible rating -- they are usually skewed to the positive end, because people are pretty good at selecting what they read or view. The solution presented will tend to skew low-frequency ratings too far toward the average. It should be easy to calculate an expected average for your site, and use this instead of 1/2 for the formula given in the post.

Major: What you really need is a missing data point: the number of readers who haven't made any rating at all. It's pretty clear if you think of two examples:

Item 1 1,000,000 views 10 positive ratings 1 negative rating

Item 2 15 views 10 positive ratings 1 negative rating

Clearly, readers feel pretty blah about Item 1. You're much better off displaying Item 2, other things being equal, but the formula doesn't distinguish the two cases.

Re: How Not To Sort By Average Rating

#13

Post by yish »

JustSemantics wrote: post #12

It's very nice to see a mathematically motivated piece on how to sort based on users ratings. But there are two problems with the solution presented, the first minor, and the second more serious. Minor: Distributions of ratings are not usually normal about the average possible rating -- they are usually skewed to the positive end, because people are pretty good at selecting what they read or view. The solution presen…

I agree, especially that the skew needs to be site dependent. For example the iPhone app store actually has the opposite problem where their rate on delete "feature", skews the average ratings way down. Reviews are heavily distributed along 1 star or 5 star ratings.

Would also be curious to see a comparison of different rating systems, for example i'd like to see what the distribution is on a site thumbs up/down rating site like digg vs a 5 star system.

Re: How Not To Sort By Average Rating

#14

Post by frederickcook »

Automatt-2 wrote: post #6

This is a very helpful blog post from about a year ago. We implemented something similar to a Wilson score to power the default sorting for the millions of reviews we have at RateItAll. In general, with a system like this, the items with the most reviews are going to rise to the top. This might not be what you want. You can easily see this effect with something like our "Events of 2010" list: http://www.rateitall.com…

A couple of notes there: first the item at the top of the list has more than an order of magnitude more ratings than the rest. For most ordered lists, this is probably an edge case that may have to be dealt with separately.

Finally, I've got to say, 26 reviews (for the #2 item) doesn't seem insignificant, and the #2 item seems to have something like a 15% higher _average_ rating. Also, on another trending rating (http://www.rateitall.com/t-3239938-2010-ncaa-tournament-team...), the #2 has a significantly higher rating (17-18% it looks like), but only one less review: 4 vs. 3. I think if I were sorting either of these based on the average rating and the # of ratings, I would have done both differently.

Based on this, it seems that the Wilson score probably _over-emphasizes_ sample size, especially on things like ratings on high-traffic internet sites that may have orders of magnitude swings in the number of ratings.

It seems like the Amazon method of average works just fine, except for items with very few ratings possibly receiving disproportionately high ratings. Again, edge cases, which should probably just be penalized manually.

Re: How Not To Sort By Average Rating

#16

Post by JustSemantics »

yish wrote: post #13
JustSemantics wrote: post #12

It's very nice to see a mathematically motivated piece on how to sort based on users ratings. But there are two problems with the solution presented, the first minor, and the second more serious. Minor: Distributions of ratings are not usually normal about the average possible rating -- they are usually skewed to the positive end, because people are pretty good at selecting what they read or view. The solution presen…

I agree, especially that the skew needs to be site dependent. For example the iPhone app store actually has the opposite problem where their rate on delete "feature", skews the average ratings way down. Reviews are heavily distributed along 1 star or 5 star ratings. Would also be curious to see a comparison of different rating systems, for example i'd like to see what the distribution is on a site thumbs up/down rati…

One data point is the netflix rating set from their contest. The system is 1 to 5 stars. The average movie rating was 3.4 in 1998 climbing to 3.8 in 2005.

Back to the original point -- if you used a formula with a default rating of 3 it would underestimate low-frequency movies. People don't view movies at random. They try to pick things they will like, which should be reflected in the statistical model.

Re: How Not To Sort By Average Rating

#17

Post by tel »

gjm11 wrote: post #5

The wrong solutions the article describes are certainly wrong. It doesn't really make much of a case that the particular solution it proposes is particularly good. For instance, here's another much simpler formula that avoids the problems described in the article: score = (pos+1)/(pos+neg+2). It's the posterior mean of Pr(random person likes the product), if your prior is uniform on [0,1]. You can adjust this to favo…

If we're going to use a Bayesian treatment, the real sin is throwing away your uncertainty.

What I would love to see is a ranking algorithm that provides an elegant, intuitive interface to fuzzy ranking. If we can't be confident of the ordering, why pretend? It's a usability problem, not a statistics problem.

Your estimator is slightly more resilient than Amazon's and somewhat less conservative than the author's. It's definitely simpler, but who cares? Visitors never have to compute it after all.

Re: How Not To Sort By Average Rating

#18

Post by ck2 »

I use what reddit uses, works well

http://code.reddit.com/browser/r2/r2/lib/db/sorts.py

      s = score(ups, downs)
      order = log(max(abs(s), 1), 10)
      sign = 1 if s > 0 else -1 if s
Let me know if you want my attempt at a MySQL translation of that.

ps. why is my account ( _ck_ ) so slow when logged in and my comments marked as "dead" ? Was I flagged as a spammer by mistake? Example: http://news.ycombinator.com/item?id=1219475

Re: How Not To Sort By Average Rating

#19

Post by araneae »

tel wrote: post #17
gjm11 wrote: post #5

The wrong solutions the article describes are certainly wrong. It doesn't really make much of a case that the particular solution it proposes is particularly good. For instance, here's another much simpler formula that avoids the problems described in the article: score = (pos+1)/(pos+neg+2). It's the posterior mean of Pr(random person likes the product), if your prior is uniform on [0,1]. You can adjust this to favo…

If we're going to use a Bayesian treatment, the real sin is throwing away your uncertainty. What I would love to see is a ranking algorithm that provides an elegant, intuitive interface to fuzzy ranking. If we can't be confident of the ordering, why pretend? It's a usability problem, not a statistics problem. Your estimator is slightly more resilient than Amazon's and somewhat less conservative than the author's. It'…

Agreed. Even if you find the formula scary, he implemented it in Ruby for you. It's a pretty short trip to implement it in the language of your choice.

Re: How Not To Sort By Average Rating

#20

Post by araneae »

ck2 wrote: post #18

I use what reddit uses, works well http://code.reddit.com/browser/r2/r2/lib/db/sorts.py s = score(ups, downs) order = log(max(abs(s), 1), 10) sign = 1 if s > 0 else -1 if s Let me know if you want my attempt at a MySQL translation of that. ps. why is my account ( _ck_ ) so slow when logged in and my comments marked as "dead" ? Was I flagged as a spammer by mistake? Example: http://news.ycombinator.com/item?id=1219475

That's not really appropriate for what this blog post was talking about, which is rating products. It doesn't make sense for products' ratings to be time sensitive. That would make the most recent items have an artificially inflated score.
Post reply on HN